Image picking-up processing device, image picking-up device, image processing method and computer program

ABSTRACT

An image processing device and an image processing method capable of generating a moving image having a high resolution and a high frame rate are provided by suppressing the reduction in the amount of incident light on each camera. The imaging and processing device includes a separation section for separating visible light into at least a first color component and a second color component; a first imaging section for taking a moving image of the first color component, wherein the first imaging section takes images of the moving image with a first spatial resolution and a first temporal resolution by exposure for a first charge accumulation time period; a second imaging section for taking a moving image of the second color component, wherein the second imaging section takes images of the moving image with a second spatial resolution higher than the first spatial resolution and a second temporal resolution lower than the first temporal resolution by exposure for a second charge accumulation time period longer than the first charge accumulation time period; a control section for controlling imaging conditions of the first and the second imaging sections; and a processing section for generating a moving image of the second component having the temporal and spatial resolutions thereof increased, based on information on the moving image of each of the first color component and the second color component.

TECHNICAL FIELD

The present invention relates to image processing of moving images, andmore specifically to a technology for generating a moving image obtainedby increasing at least one of the resolution and the frame rate of animaged moving image by image processing.

BACKGROUND ART

Recently in the field of video input, the number of pixels of camerasfor mobile phones and digital still cameras is increasing and the pixelpitch thereof is decreasing.

Various spatial resolutions are used for different levels of imagequality required for various imaging devices. For example, theresolution of TV phones is of a relatively small number of pixels; forexample, approximately QCIF (Quarter Common Intermediate Format; 176pixels horizontally and 144 pixels vertically) or approximately QVGA(Quarter Video Graphics Array, 320 pixels horizontally and 144 pixelsvertically). By contrast, the resolution of digital single-lens reflexcameras exceeds 10 million pixels.

Various temporal resolutions are also used for different levels of imagequality. For example, regarding the temporal resolution of up to thenumber of pixels of HDTVs, imaging is performed at the video rate ofconsumer devices (30 frames/sec.). By contrast, for performing imagingat a greater number of pixels, the frame rate is merely several framesper second at the maximum, which is realized by the consecutive shootingfunction provided by digital still cameras.

Meanwhile, in the field of video display, flat TVs are rapidlyspreading. In accordance with this, users are expected to view videomaterials by a combination of a camera and a display of variousresolutions in the future.

Comparing the temporal and spatial resolutions of the camera on theinput side (“temporal and spatial resolutions” means temporal resolutionand spatial resolution; the same is also applied below) and the temporaland spatial resolutions of the display on the output side, the temporaland spatial resolutions of the display on the output side are higher inthe currently available consumer devices. Therefore, general users todaycannot easily obtain a video material which allows the device on theoutput side to make the maximum use of the performance thereof.

A reason why such a situation has occurred is that the reading speed isconventionally the bottleneck. The imaging at a high spatial resolutionis limited to being performed at 5 frames per second, whereas theimaging at 30 frames per second is limited to being performed at thespatial resolution of HDTVs. For this reason, it is conventionallydifficult to perform imaging at a high spatial resolution and a highframe rate.

In order to address the above-described problems, Patent Documents 1through 3, for example, propose a method for providing both a highspatial resolution and a high frame rate, by which images havingdifferent temporal resolutions and different spatial resolutions areinput from cameras of two systems to generate an image having a highspatial resolution and a high frame rate by signal processing. Thesepatent documents describe a structure shown in FIG. 19.

FIG. 19 shows a structure of a conventional imaging device. Incidentlight on the imaging device is partially transmitted through a halfmirror 171 and incident on a first camera 172. As a result, a movingimage having a low resolution and a high frame rate is taken. Lightwhich is incident on the imaging device and reflected by the half mirror171 is incident on a second camera 173. As a result, a moving imagehaving a high resolution and a low frame rate is taken.

An upconverter 174 receives the moving images taken by the first camera172 and the second camera 173 and performs image processing to output amoving image having a high resolution and a high frame rate.

Patent Document 1: Japanese Laid-Open Patent Publication No. 7-143439

Patent Document 2: PCT Japanese National Phase Laid-Open PatentPublication No. 2005-515675

Patent Document 3: Japanese Laid-Open Patent Publication No. 2005-318548

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

However, in the above-described imaging device, the light is dividedinto two by the half mirror 171, and therefore the amount of light maybe occasionally insufficient at the second camera 173 for higherresolution imaging, which has a smaller pixel size. In addition, theamount of light may be occasionally insufficient also at the firstcamera 172 for lower resolution imaging, for which an upper limit is setfor the exposure time for taking a moving image. This lowers theluminance of the image, resulting in a dark image.

The present invention for solving the above-described problems has anobject of providing an image processing device and an image processingmethod capable of preventing the amount of incident light on each ofcameras from being reduced and generating a moving image having both ahigh resolution and a high frame rate.

Means for Solving the Problems

An imaging and processing device according to the present inventioncomprises a separation section for separating visible light into atleast a first color component and a second color component; a firstimaging section for taking a moving image of the first color component,the first imaging section taking images which form the moving image witha first spatial resolution and a first temporal resolution by exposurefor a first charge accumulation time period; a second imaging sectionfor taking a moving image of the second color component, the secondimaging section taking images which form the moving image with a secondspatial resolution higher than the first spatial resolution and a secondtemporal resolution lower than the first temporal resolution by exposurefor a second charge accumulation time period longer than the firstcharge accumulation time period; a control section for controllingimaging conditions of the first imaging section and the second imagingsection; and a processing section for generating a moving image of thesecond component having the temporal and spatial resolutions thereofincreased, based on information on the moving image of the first colorcomponent and information on the moving image of the second colorcomponent.

An imaging and processing device according to the present inventioncomprises a separation section for separating visible light into atleast a first color component and a second color component; a firstimaging section for taking a moving image of the first color component,the first imaging section taking images which form the moving image witha first spatial resolution and a first temporal resolution by exposurefor a first charge accumulation time period; a second imaging sectionfor taking a moving image of the second color component, the secondimaging section taking images which form the moving image with a secondspatial resolution higher than the first spatial resolution and a secondtemporal resolution lower than the first temporal resolution by exposurefor a second charge accumulation time period longer than the firstcharge accumulation time period; a motion detection section forgenerating motion information based on a time-wise change of the movingimage of the first color component; and a processing section forgenerating a moving image of the second component having the temporaland spatial resolutions thereof increased, based on information on themoving image of the first color component, information on the movingimage of the second color component and the motion information.

The processing section may hold a relational expression of the movingimage of the second color component taken by the second imaging sectionand the moving image of the second component having the spatialresolution thereof increased; and the processing section may generatethe moving image of the second component having the spatial resolutionthereof increased, in accordance with the degree at which the relationalexpression is fulfilled, using, as constraining conditions, temporal andspatial correspondence between the moving image of the first colorcomponent and the moving image of the second color component, acondition for smoothness between pixels close to each other in theimage, an assumption that the brightness of an imaging subject moving inthe image is constant, and a local correlation between pixel values ofthe first color component and the second color component.

The processing section may output a moving image, which fulfills therelational expression best under the moving image of the second colorcomponent taken by the second imaging section and the constrainingconditions, as the moving image of the second component having thespatial resolution thereof increased.

The processing section may hold a relational expression of the movingimage of the second color component taken by the second imaging sectionand the moving image of the second component having the spatialresolution thereof increased; and the processing section may output amoving image fulfilling the relational expression as the moving image ofthe second component having the spatial resolution thereof increased,using, as constraining conditions, temporal and spatial correspondencebetween the moving image of the first color component and the movingimage of the second component, and a condition for smoothness betweenpixels close to each other in the image.

The processing section may hold a relational expression of the movingimage of the second color component taken by the second imaging sectionand the moving image of the second component having the spatialresolution thereof increased; and the processing section may output amoving image fulfilling the relational expression as the moving image ofthe second component having the spatial resolution thereof increased,using, as constraining conditions, temporal and spatial correspondencebetween the moving image of the first color component and the movingimage of the second color component, a condition for smoothness betweenpixels close to each other in the image, and an assumption that thebrightness of an imaging subject moving in the image is constant.

The imaging and processing device may further comprise a third imagingsection. The separation section may separate the visible light into thefirst color component, the second color component and a third colorcomponent; the third imaging section may take images which form a movingimage of the third color component with a third spatial resolution and athird temporal resolution by exposure for a third charge accumulationtime period; and the third charge accumulation time period may beshorter than the second charge accumulation time period, the thirdspatial resolution may be lower than the second spatial resolution, andthe third temporal resolution may be higher than the second temporalresolution.

The processing section may hold a relational expression of the movingimage of the second color component taken by the second imaging sectionand the moving image of the second component having the spatialresolution thereof increased; and the processing section may output amoving image fulfilling the relational expression as the moving image ofthe second component having the spatial resolution thereof increased,using, as constraining conditions, temporal and spatial correspondencebetween the moving image of the first color component, the moving imageof the second color component and the moving image of the third colorcomponent, and a local correlation among pixel values of the first colorcomponent, the second color component and the third color component.

The processing section may hold a relational expression of the movingimage of the second color component taken by the second imaging sectionand the moving image of the second component having the spatialresolution thereof increased; and the processing section may output amoving image fulfilling the relational expression as the moving image ofthe second component having the spatial resolution thereof increased,using, as constraining conditions, temporal and spatial correspondenceamong the moving image of the first color component, the moving image ofthe second color component and the moving image of the third colorcomponent, a condition for smoothness between pixels close to each otherin the image, an assumption that the brightness of an imaging subjectmoving in the image is constant, and a local correlation among pixelvalues of the first color component, the second color component and thethird color component.

An imaging device according to the present invention forms an imageprocessing system together with an image processing device. The imagingdevice comprises a separation section for separating visible light intoat least a first color component and a second color component; a firstimaging section for taking a moving image of the first color component,the first imaging section taking images which form the moving image witha first spatial resolution and a first temporal resolution by exposurefor a first charge accumulation time period; a second imaging sectionfor taking a moving image of the second color component, the secondimaging section taking images which form the moving image with a secondspatial resolution higher than the first spatial resolution and a secondtemporal resolution lower than the first temporal resolution by exposurefor a second charge accumulation time period longer than the firstcharge accumulation time period; and a control section for controllingimaging conditions of the first imaging section and the second imagingsection. The imaging device outputs information on the moving image ofthe first color component and information on the moving image of thesecond color component in order to allow the image processing device togenerate a moving image of the second component having the temporalresolution thereof increased.

The imaging device may further comprise a third imaging section. Theseparation section may separate the visible light into the first colorcomponent, the second color component and a third color component; thethird imaging section may take images which form a moving image of thethird color component with a third spatial resolution and a thirdtemporal resolution by exposure for a third charge accumulation timeperiod; and the third charge accumulation time period may be shorterthan the second charge accumulation time period, the third spatialresolution may be lower than the second spatial resolution, and thethird temporal resolution may be higher than the second temporalresolution.

An image processing device according to the present invention is animage processing device for receiving information on a moving image froman imaging device and processing the information. The imaging deviceincludes a separation section for separating visible light into at leasta first color component and a second color component; a first imagingsection for taking a moving image of the first color component, thefirst imaging section taking images which form the moving image with afirst spatial resolution and a first temporal resolution by exposure fora first charge accumulation time period; a second imaging section fortaking a moving image of the second color component, the second imagingsection taking images which form the moving image with a second spatialresolution higher than the first spatial resolution and a secondtemporal resolution lower than the first temporal resolution by exposurefor a second charge accumulation time period longer than the firstcharge accumulation time period; and a control section for controllingimaging conditions of the first imaging section and the second imagingsection. The image processing device comprises a motion detectionsection for receiving information on the moving image of the first colorcomponent and information on the moving image of the second colorcomponent and generating motion information based on a time-wise changeof the moving image of the first color component; and a processingsection for generating a moving image of the second component having thespatial resolution thereof increased, based on the information on themoving image of the first color component, the information on the movingimage of the second color component and the motion information.

An image processing method according to the present invention comprisesthe steps of receiving, from the above-described imaging device, theinformation on the taken moving image of the first color component andthe information on the taken moving image of the second color component;generating motion information based on a time-wise change of the movingimage of the first color component; obtaining a relational expression ofthe moving image of the second color component and a moving image of thesecond component having a spatial resolution thereof increased; andgenerating the moving image of the second component having the spatialresolution thereof increased, in accordance with the degree at which therelational expression is fulfilled, using, as constraining conditions,temporal and spatial correspondence between the moving image of thefirst color component and the moving image of the second colorcomponent, a condition for smoothness between pixels close to each otherin the image, and an assumption that the brightness of an imagingsubject moving in the image is constant.

The step of generating the moving image of the second component maygenerate the moving image of the second component having the spatialresolution thereof increased, in accordance with the degree at which therelational expression is fulfilled, further using, as a constrainingcondition, a local correlation between the pixel values of the firstcolor component and the second color component.

An image processing method according to the present invention comprisesthe steps of receiving, from the above-described imaging device, theinformation on the taken moving image of the first color component andthe information on the taken moving image of the second color component;obtaining a relational expression of the moving image of the secondcolor component and a moving image of the second component having aspatial resolution thereof increased; and generating the moving image ofthe second component having the spatial resolution thereof increased, inaccordance with the degree at which the relational expression isfulfilled, using, as constraining conditions, temporal and spatialcorrespondence between the moving image of the first color component andthe moving image of the second color component, and a condition forsmoothness between pixels close to each other in the image.

An image processing method according to the present invention comprisesthe steps of receiving, from the above-described imaging device, theinformation on the taken moving image of the first color component andthe information on the taken moving image of the second color component;obtaining a relational expression of the moving image of the secondcolor component and a moving image of the second component having aspatial resolution thereof increased; and generating the moving image ofthe second component having the spatial resolution thereof increased, inaccordance with the degree at which the relational expression isfulfilled, using, as constraining conditions, temporal and spatialcorrespondence between the moving image of the first color component andthe moving image of the second color component, and a local correlationbetween the pixel values of the first color component and the secondcolor component.

An image processing method according to the present invention comprisesthe steps of receiving, from the above-described imaging device, theinformation on the taken moving images of the first color component, thesecond color component and the third color component; obtaining arelational expression of the moving image of the second color componentand a moving image of the second component having a spatial resolutionthereof increased; and outputting a moving image fulfilling therelational expression as the moving image of the second component havingthe spatial resolution thereof increased, using, as constrainingconditions, temporal and spatial correspondence among the moving imageof the first color component, the moving image of the second colorcomponent and the moving image of the third color component, a conditionfor smoothness between pixels close to each other in the image, anassumption that the brightness of an imaging subject moving in the imageis constant, and a local correlation among the pixel values of the firstcolor component, the second color component and the third colorcomponent.

A computer program according to the present invention is a computerprogram for causing a processor included in an image processing deviceto generate data on a moving image based on data on a first moving imageand data on a second moving image. The computer program causes theprocessor to execute the steps of receiving, from the above-describedimaging device, the information on the taken moving images of the firstcolor component, the second color component and the third colorcomponent; reading a relational expression of the moving image of thesecond color component and a moving image of the second component havinga spatial resolution thereof increased; and generating a moving image ofthe second component having a spatial resolution thereof increased, inaccordance with the degree at which the relational expression isfulfilled, using, as constraining conditions, temporal and spatialcorrespondence between the moving image of the first color component andthe moving image of the second color component, a condition forsmoothness between pixels close to each other in the image, anassumption that the brightness of an imaging subject moving in the imageis constant, and a local correlation between the pixel values of thefirst color component and the second color component.

The computer program may be recorded on a recording medium.

EFFECTS OF THE INVENTION

An imaging and processing device according to the present inventionmakes it possible to generate a multi-color moving image having both ahigh resolution and a high frame rate from moving images of a pluralityof color components having different resolutions and different framerates. Each of the moving images of the plurality of color components istaken by separating the incident light color by color using, forexample, a dichroic mirror, without using a half mirror or the like.Owing to this, unlike the conventional art using a half mirror, theamount of light received by each imaging section is not reduced to half.Therefore, the amount of light of the moving image of each colorcomponent is not reduced, and so a bright moving image is provided.Based on such color component images, a moving image having both a highresolution and a high frame rate is generated. The moving image thusobtained is also bright.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a structure of an imaging and processing device accordingto Embodiment 1.

FIG. 2 shows a structure of an upconverter 106 for G in more detail.

FIGS. 3( a) and (b) respectively show a baseline frame and a referenceframe used for performing motion detection using block matching.

FIG. 4 shows a group of R, G and B pixels forming a color image of 2×2pixels.

FIG. 5 shows pixels i, i+1, i+w, and i+w+1 which form a local area in alarger image.

FIG. 6 shows the relationship between the local correlation ρ between Rand B with the weight W.

FIG. 7 shows a structure of an upconverter 107 for R and B in moredetail.

FIG. 8 shows a structure of an upconverter 107 for R and B in anotherexample in detail.

FIG. 9 shows an example of the relationship between the correlationvalue ρ and the weight W2.

FIG. 10 shows a structure of an upconverter 107 for R and B in amodification.

FIG. 11 shows a structure of an imaging and processing device 11according to a modification of Embodiment 1.

FIG. 12 shows a structure of an imaging and processing device 12according to another modification of Embodiment 1.

FIG. 13 shows a structure of an imaging and processing device 13, whichis obtained by as a result of generalizing the imaging and processingdevice 1.

FIG. 14 shows an example of an image processing system according toEmbodiment 2 including an imaging device 901, a network 902 and aprocessing device 903.

FIG. 15 shows hardware of an image processing device constructed by acomputer.

FIG. 16 is a flowchart showing a procedure of processing according tothe present invention.

FIG. 17 is a flowchart showing a detailed procedure of the processing instep S103 shown in FIG. 16.

FIG. 18 is a flowchart showing a detailed procedure of processing instep S103 in another example.

FIG. 19 shows a structure of a conventional imaging device.

FIG. 20 shows a group of R, G and B pixels forming a color image of2×2×2 pixels.

FIG. 21 shows a group of R, G and B pixels forming a color image of apixel of interest and three pixels in the vicinity thereof.

DESCRIPTION OF THE REFERENCE NUMERALS

-   -   101 Lens system    -   102 Dichroic mirror    -   103 First imaging section    -   104 Second imaging section    -   105 Image processing section    -   106 Upconverter for G    -   107 Upconverter for R and B    -   120 Control section

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of an imaging and processing device accordingto the present invention will be described with reference to theattached drawings.

Embodiment 1

FIG. 1 shows a structure of an imaging and processing device 1 accordingto this embodiment. The imaging and processing device 1 includes a lenssystem 101, a dichroic mirror 102, a first imaging section 103, a secondimaging section 104, an image processing section 105, and a controlsection 120.

In the following, a general function of each element will be firstdescribed, and then an operation of each element will be described indetail in relation with an operation of the imaging and processingdevice 1.

The lens system 101 converges incident light from outside the imagingand processing device 1, namely, an image of an imaging subject.

The dichroic mirror 102 allows red (R) and blue (B) components of thelight to be transmitted therethrough and reflects a green (G) componentof the light. Namely, the dichroic mirror 102 separates the incidentlight into the red (R) and blue (B) components and the green (G)component. Hereinafter, the red component will also be referred to asthe “R component”, the green component will also be referred to as the“G component”, and the blue component will also be referred to as the “Bcomponent”.

The first imaging section 103 takes a moving image of each of the Rcomponent and the B component with a short-time exposure, a lowresolution and a high frame rate based on the incident light (here, theR component and the B component of the light), and outputs the resultantdata. In order to obtain the moving image of the R component and themoving image of the B component, the first imaging section 103 may havea dichroic mirror inside thereof and may also include an imaging elementfor detecting the R component and the B component.

The second imaging section 104 takes a moving image of the G componentwith a long-time exposure, a high resolution and a low frame rate basedon the incident light (here, the G component of the light), and outputsdata on the moving image of the G component.

The image processing section 105 receives the data on the moving imagescorresponding to the light of the R component and the B component andthe data on the moving image corresponding to the light of the Gcomponent. The image processing section 105 converts the data on eachmoving image into a moving image having a high resolution and a highframe rate by image processing, and outputs the resultant data.

The image processing section 105 includes an upconverter 106 for G andan upconverter 107 for R and B.

The upconverter 106 for G generates data on a moving image having a highresolution and a high frame rate as a result of increasing the framerate of the G component. The upconverter 107 for R and B generates dataon moving images having a high resolution and a high frame rate as aresult of increasing the resolutions of the R component and the Bcomponent. The moving images are displayed by switching one or aplurality of images consecutively at a prescribed frame rate. Theprocessing performed by the upconverter 107 for R and B for increasingthe resolutions means increasing the number of pixels of each of theimages forming the moving images.

The upconverter 106 for G and the upconverter 107 for R and B will bedescribed later in detail.

The control section 120 controls the imaging conditions for taking themoving images using the first imaging section 103 and the second imagingsection 104. The control section 120 outputs the control information tothe upconverter 106 for G and the upconverter 107 for R and B.

Now, an operation of each element will be described together with anoperation of the imaging and processing device 1.

The position of the lens system 101 is adjusted such that images of theimaging subject are formed on imaging elements of the first imagingsection 103 and the second imaging section 104. The light which haspassed through the lens system 101 is separated into an R component, a Gcomponent and a B component by the dichroic mirror 102.

The moving images of the R component and the B component are taken bythe first imaging section 103 with the imaging conditions instructed bythe control section 120, namely, with a short-time exposure, a lowresolution and a high frame rate. Here, “low resolution” means, forexample, the resolution of approximately the number of pixels of oneframe of NTSC (720 pixels horizontally×480 pixels vertically) or a lowerresolution of approximately the number of pixels of VGA (Video GraphicsArray: 640 pixels horizontally×480 pixels vertically). “High frame rate”means about 30 fps (frames/sec.) through 60 fps. “Short-time exposure”means exposure for a time period of an upper limit determined by theframe rate (in this embodiment, 1/30 sec. through 1/60 sec.) at thelongest.

The moving image of the G component is taken by the second imagingsection 104, again with the imaging conditions instructed by the controlsection 120, namely, with a long-time exposure, a high resolution and alow frame rate. Here, “high resolution” means, for example, theresolution of the number of pixels of a general digital still camera(for example, 4000 pixels horizontally and 3000 pixels vertically). “Lowframe rate” means a few tens of percent or 1/10 through 1/20 of that ofthe first imaging section 103 (for example, 3 fps (frames/sec.).“Long-time exposure” means exposure for a time period determined by thevalue of the low frame rate (for example, 1/3 sec.) at the longest.

In this embodiment, the first imaging section 103 and the second imagingsection 104 operate while being synchronized to each other by thecontrol section 120. However, it is not indispensable that the firstimaging section 103 and the second imaging section 104 are synchronizedto each other.

The above-described long/short-time exposure, high/low resolution andhigh/low frame rate are relative imaging conditions of the first imagingsection 103 and the second imaging section 104. It is sufficient thatthe exposure time of each of the R component and the B component of acolor image is shorter than that of the G component. It is sufficientthat the resolution (here, corresponding to the number of pixels) ofeach of the R component and the B component is lower than that of the Gcomponent. It is sufficient that the frame rate of each of the Rcomponent and the B component is higher than that of the G component.The above-described ranges of numerical values are examples and notlimiting.

Hereinafter, in this specification, a moving image of the G colorcomponent having a high (H) resolution and a low (L) frame rate will berepresented as G_(HL), and moving images of the R and B color componentshaving a low (L) resolution and a high (H) frame rate will berepresented respectively as R_(LH) and B_(LH). The first letterrepresents the color component, the second letter (first subscript)represents the resolution, and the third letter (second subscript)represents the frame rate.

The upconverter 106 for G receives data on the G image G_(HL) taken witha long-time exposure, a high resolution and a low frame rate and data onthe R image R_(LH) and the B image B_(LH) taken with a short-timeexposure, a low resolution and a high frame rate as the imagingconditions, and outputs G_(HH) as a result of increasing the resolutionof G_(HL).

Namely, the upconverter 106 for G generates a synthesized moving imageas a result of increasing the frame rate while keeping the resolution.This means that the processing is performed such that the greensynthesized moving image has the best subjective image quality.

The reason for performing the processing in this manner is that humansight has a feature of being more sensitive to green than to red or blueand therefore it is generally desirable that the subjective imagequality of the green synthesized moving image is best.

In order to improve the subjective image quality of the greensynthesized moving image, it is often considered preferable to take agreen moving image with a high resolution and a low frame rate. Forexample, it is assumed that the imaging subject in an image is still oris moving only a little. In such a case, where the green moving image istaken with a high resolution and a low frame rate, the resolution of thegreen synthesized moving image is higher than that of the red or bluesynthesized moving image and as a result, the subjective quality of theentire image is improved. It is also expected that the subjective imagequality is further improved by increasing the frame rate of the greenmoving image as compared with the frame rate of the red or blue movingimage.

FIG. 2 shows a structure of the upconverter 106 for G in more detail. InFIG. 2, elements common to those in the imaging and processing deviceshown in FIG. 1 bear identical reference numerals thereto anddescriptions thereof will be omitted.

The upconverter 106 for G includes a motion detection section 108 and atemporal resolution upconverter 109.

The motion detection section 108 detects a motion (optical flow) fromR_(LH) and B_(LH) by an existing known technology such as a blockmatching method, a gradient method, a phase correlation method or thelike. One such known technology is described in, for example, J. L.Barron, D. J. Fleet, S. S. Beauchemin, and T. A. Burkitt, “Performanceof Optical Flow Techniques”, In Proc. Computer Vision and PatternRecognition, pp. 236-242, 1992.

FIG. 3( a) and (b) respectively show a baseline frame and a referenceframe used for performing motion detection using block matching. Themotion detection section 108 sets a window area A shown in FIG. 3( a) ina frame used as a baseline (image at time t of interest at which themotion is to be detected), and searches in the reference frame for apattern similar to the pattern in the window area. As the referenceframe, a frame next to the frame of interest is often used, for example.

Usually as shown in FIG. 3( b), a certain range (C in FIG. 3( b)) ispreset as the search range, based on position B at which the movingamount is zero. The degree (extent) of similarity of the patterns isevaluated by calculating the sum of square differences (SSD) shown inexpression 1 or the sum of absoluted differences (SAD) shown inexpression 2 as the evaluation value.

$\begin{matrix}{{SSD} = {\sum\limits_{x,{y \in W}}\left( {{I\left( {{x + u},{y + v},{t + {\Delta \; t}}} \right)} - {I\left( {x,y,t} \right)}} \right)^{2}}} & \left\lbrack {{Expression}\mspace{14mu} 1} \right\rbrack \\{{SAD} = {\sum\limits_{x,{y \in W}}{{{I\left( {{x + u},{y + v},{t + {\Delta \; t}}} \right)} - {I\left( {x,y,t} \right)}}}}} & \left\lbrack {{Expression}\mspace{14mu} 2} \right\rbrack\end{matrix}$

In expressions 1 and 2, x,yεW means the coordinate value of the pixelencompassed in the window area of the baseline frame.

The motion detection section 108 changes (u, v) in the search range tosearch for a (u, v) set at which the above-described evaluation value isminimum, and uses the obtained (u, v) set as an inter-frame motionvector V. By sequentially shifting the position of the window area, themotion is obtained pixel by pixel or block by block (for example, 8pixels×8 pixels).

FIG. 2 is referred to again. The temporal resolution upconverter 109receives data on each of the R image R_(LH) and the B image B_(LH) takenby the first imaging section 103, data on the G image G_(HL) taken bythe second imaging section 104, and data on the motion vector V detectedby the motion detection section 108, outputs G_(HH) as a result ofincreasing the resolution of G.

The resolution of G_(HL) is increased by obtaining G_(HH) whichminimizes expression 3 below.

J=(H _(T) G _(HH) −G _(HL))²+λ_(s)(Q _(s) G _(HH))²+λ_(m)(Q _(m) G_(HH))²+λ_(c)(Q _(c) H _(s) G _(HH))²,  [Expression 3]

where G_(HH) and G_(HL) represent vertical vectors having the pixels ofthe moving image as the elements, H_(T) represents the matrix modelingthe addition of light by long-time exposure, λ_(s) represents the weightto the smoothness constraint, Q_(s) represents the smoothnessconstraint, λ_(m) represents the weight to the motion constraint, Q_(m)represents the motion constraint, λ_(c) represents the weight to thecolor correlation constraint, H_(s) represents the matrix modeling therelationship between a high resolution image and a low resolution image,and Q_(c) represents the color correlation constraint. The imagingconditions set for the second imaging section 104 by the control section120 are reflected on at least H_(T), Q_(s), Q_(m) and Q_(c).

G_(HH) which minimizes expression 3 means G_(HH) which most satisfiesthe linear sum of the given constraining conditions. The left side ofexpression 3 is scalar. The process of deriving the right side will bedescribed later.

The temporal resolution upconverter 109 obtains G_(HH) which minimizesexpression 3 based on expression 4 below.

$\begin{matrix}{\frac{\partial J}{\partial G_{HH}} = {{{2{H_{T}^{T}\left( {{H_{T}G_{HH}} - G_{HL}} \right)}} + {2\lambda_{s}Q_{s}^{T}Q_{s}G_{HH}} + {2\lambda_{m}Q_{m}^{T}Q_{m}G_{HH}} + {2\lambda_{c}H_{s}^{T}Q_{c}^{T}Q_{c}H_{s}G_{HH}}} = 0}} & \left\lbrack {{Expression}\mspace{14mu} 4} \right\rbrack\end{matrix}$

As a result, the temporal resolution upconverter 109 obtains G_(HH) bysolving the simultaneous equations represented by expression 5.

(H _(T) ^(T)+λ_(s) Q _(s) ^(T) Q _(s)+λ_(m) Q _(m) ^(T) Q _(m)+λ_(c) H_(s) ^(T) Q _(c) ^(T) Q _(c) H _(s))G _(HH) =H _(T) ^(T) G_(HL)  [Expression 5]

Expression 5 can be solved by an existing numerical value calculationmethod (method for solving simultaneous equations) such as conjugategradient method, steepest descent method or the like.

The processing performed by the temporal resolution upconverter 109 forincreasing the resolution of the input G_(HL) by the above-describedprocedure to obtain G_(HH) includes processing of increasing the numberof frames, namely, processing of increasing the temporal resolution, andalso processing of converting a blurred image to a clear (sharp) image.The moving image of the G component is taken by the second imagingsection 104 with a high resolution, but a part thereof may be possiblyblurred by the influence of the long-time exposure. Hence, the entireprocessing including the processing of sharpening the image is referredto as “increasing the resolution”.

Hereinafter, the meaning and function of each term of expression 3 willbe described in more detail.

The first term of expression 3 indicates the difference between thelong-time exposure image expected from the G_(HH) with the increasedresolution and the actually observed long-time exposure image G_(HL).This represents the temporal and spatial correspondence between G_(HH)with the increased resolution and the long-time exposure image G_(HL).Here, at H_(T), the number of rows is smaller than the number ofcolumns. This is also understood from that G_(HL) is a long-timeexposure image and has a lower frame rate than that of G_(HH) (namely,the total number of pixels of G_(HL) is smaller than that of G_(HH)).Therefore, if the left side only includes the first term, expression 3is an ill-posed problem, by which the problem to be solved (namely, thesimultaneous expressions) cannot be solved uniquely.

In order to change this ill-posed problem into a well-posed problem, thesecond and the other terms are added to the first term of expression 3.The second term of expression 3 indicates the characteristic generallyfulfilled by the image, namely, the local smoothness between pixelslocated close to each other. Where the moving image is I(x, y, t), thelocal smoothness can be represented as:

$\begin{matrix}{{\int{{{\nabla^{2}{I\left( {x,y,t} \right)}}}^{2}{x}{y}{t}}} = {\int{{\begin{pmatrix}{\frac{\partial^{2}}{\partial x^{2}}{I\left( {x,y,t} \right)}} \\{\frac{\partial^{2}}{\partial y^{2}}{I\left( {x,y,t} \right)}} \\{\frac{\partial^{2}}{\partial t^{2}}{I\left( {x,y,t} \right)}}\end{pmatrix}}^{2}{x}{y}{t}}}} & \left\lbrack {{Expression}\mspace{14mu} 6} \right\rbrack\end{matrix}$

Here, the range of integration is the entire time-space occupied by themoving image. ∥·∥ in the integrand in expression 6 indicates the norm ofthe vector. The second term of expression 3 performs differenceexpansion of expression 6 and substitutes I with G to representexpression 6 in the form of the logical product of the matrix Q_(s) andthe vector G.

The third term of expression 3 is the constraint on the motion in theimage. Here, it is assumed that each point in the moving image moveswithout changing the brightness thereof. In the matrix Q_(m) in thethird term, only the elements relating to the pixels at the start pointand the end point of the motion vector detected in the moving image arerespectively 1 and −1. Therefore, (Q_(m)G_(HH))² is the total sum, inthe entire moving image, of the squares of the residual between thestart point and the end point of the motion vector.

The fourth term of expression 3 is the constraint on the localcorrelation among R, G and B. Like the local smoothness assumed for thesecond term of expression 3, local correlation (color correlation) amongthe pixel values of R, G and B is assumed for the fourth term.

Here, for simplicity, the example shown in FIG. 4 is used for theexplanation. FIG. 4 shows a group of R, G and B pixels which form onecolor image of 2×2 pixels. It is assumed that R, G and B are correlatedto one another. Namely, it is assumed that the ratio among R, G and B isuniform in all the pixels. Thus, expression 7 holds.

$\begin{matrix}{{\frac{G_{1}}{R_{1}} = {\frac{G_{2}}{R_{2}} = {\frac{G_{3}}{R_{3}} = \frac{G_{4}}{R_{4}}}}}{\frac{G_{1}}{B_{1}} = {\frac{G_{2}}{B_{2}} = {\frac{G_{3}}{B_{3}} = \frac{G_{4}}{B_{4}}}}}} & \left\lbrack {{Expression}\mspace{14mu} 7} \right\rbrack\end{matrix}$

For selecting two out of each four ratios, there are ₄C₂=6 combinations.Thus, expression 8 is obtained.

G ₁ R ₂ −G ₂ R ₁=0

G ₂ R ₃ −G ₃ R ₂=0

G ₃ R ₄ −G ₄ R ₃=0

G ₁ R ₄ −G ₄ R ₁=0

G ₁ R ₃ −G ₃ R ₁=0

G ₂ R ₄ −G ₄ R ₂=0

G ₁ B ₂ −G ₂ B ₁=0

G ₂ B ₃ −G ₃ B ₂=0

G ₃ B ₄ −G ₄ B ₃=0

G ₁ B ₄ −G ₄ B ₁=0

G ₁ B ₃ −G ₃ B ₁=0

G ₂ B ₄ −G ₄ B ₂=0  [Expression 8]

Expression 8 can be represented by a matrix and vector as expression 9.

$\begin{matrix}{{\begin{pmatrix}R_{2} & {- R_{1}} & 0 & 0 \\0 & R_{3} & {- R_{2}} & 0 \\0 & 0 & R_{4} & {- R_{3}} \\R_{4} & 0 & 0 & {- R_{1}} \\R_{3} & 0 & {- R_{1}} & 0 \\0 & R_{4} & 0 & {- R_{2}} \\B_{2} & {- B_{1}} & 0 & 0 \\0 & B_{3} & {- B_{2}} & 0 \\0 & 0 & B_{4} & {- B_{3}} \\B_{4} & 0 & 0 & {- B_{1}} \\B_{3} & 0 & {- B_{1}} & 0 \\0 & B_{4} & 0 & {- B_{2}}\end{pmatrix}\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4}\end{pmatrix}} = {{q\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4}\end{pmatrix}} = 0}} & \left\lbrack {{Expression}\mspace{14mu} 9} \right\rbrack\end{matrix}$

Where the area of 2×2 pixels is considered as a local area, thecorrelation among R, G and B is evaluated by the norm of the left sideof expression 9 (i.e., expression 10).

$\begin{matrix}{{{\begin{pmatrix}R_{2} & {- R_{1}} & 0 & 0 \\0 & R_{3} & {- R_{2}} & 0 \\0 & 0 & R_{4} & {- R_{3}} \\R_{4} & 0 & 0 & {- R_{1}} \\R_{3} & 0 & {- R_{1}} & 0 \\0 & R_{4} & 0 & {- R_{2}} \\B_{2} & {- B_{1}} & 0 & 0 \\0 & B_{3} & {- B_{2}} & 0 \\0 & 0 & B_{4} & {- B_{3}} \\B_{4} & 0 & 0 & {- B_{1}} \\B_{3} & 0 & {- B_{1}} & 0 \\0 & B_{4} & 0 & {- B_{2}}\end{pmatrix}\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4}\end{pmatrix}}}^{2} = {{{q\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4}\end{pmatrix}}}^{2} = {\begin{pmatrix}G_{1} & G_{2} & G_{3} & G_{4}\end{pmatrix}q^{T}{q\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4}\end{pmatrix}}}}} & \left\lbrack {{Expression}\mspace{14mu} 10} \right\rbrack\end{matrix}$

Here, q^(T)q can be represented by expression 11.

$\begin{matrix}{{q^{T}q} = \begin{pmatrix}\begin{matrix}{R_{2}^{2} + R_{3}^{2} + R_{4}^{2} +} \\{B_{2}^{2} + B_{3}^{2} + B_{4}^{2}}\end{matrix} & {{{- R_{1}}R_{2}} - {B_{1}B_{2}}} & {{{- R_{1}}R_{3}} - {B_{1}B_{3}}} & {{{- R_{1}}R_{4}} - {B_{1}B_{4}}} \\{{{- R_{1}}R_{2}} - {B_{1}B_{2}}} & \begin{matrix}{R_{1}^{2} + R_{3}^{2} + R_{4}^{2} +} \\{B_{1}^{2} + B_{3}^{2} + B_{4}^{2}}\end{matrix} & {{{- R_{2}}R_{3}} - {B_{2}B_{3}}} & {{{- R_{2}}R_{4}} - {B_{2}B_{4}}} \\{{{- R_{1}}R_{3}} - {B_{1}B_{3}}} & {{{- R_{2}}R_{3}} - {B_{2}B_{3}}} & \begin{matrix}{R_{1}^{2} + R_{2}^{2} + R_{4}^{2} +} \\{B_{1}^{2} + B_{2}^{2} + B_{4}^{2}}\end{matrix} & {{{- R_{3}}R_{4}} - {B_{3}B_{4}}} \\{{{- R_{1}}R_{4}} - {B_{1}B_{4}}} & {{{- R_{2}}R_{4}} - {B_{2}B_{4}}} & {{{- R_{3}}R_{4}} - {B_{3}B_{4}}} & \begin{matrix}{R_{1}^{2} + R_{2}^{2} + R_{3}^{2} +} \\{B_{1}^{2} + B_{2}^{2} + B_{3}^{2}}\end{matrix}\end{pmatrix}} & \left\lbrack {{Expression}\mspace{14mu} 11} \right\rbrack\end{matrix}$

Next, the area of 2×2 pixels will be considered as a local area in animage larger than 2×2 pixels. FIG. 5 shows pixels i, i+1, i+w and i+w+1which form the local area in the larger image. As shown in FIG. 5, wherethe top left pixel in the local area is i'th pixel, the top right pixelis (i+1)'th pixel, the bottom left pixel is the (i+w)'th pixel (here, wis the number of pixels in the width direction of the image), and thebottom right pixel is the (i+w+1)'th pixel. Therefore, expression 11 istransformed as expression 12.

$\begin{matrix}{\; {{{q_{i}^{\tau}q_{i}} = {\begin{pmatrix}\ddots & \; & \; & \; & \; & \; & \; \\\; & \begin{matrix}{R_{i + 1}^{2} + R_{i + w}^{2} +} \\{R_{i + w + 1}^{2} + B_{i + 1}^{2} +} \\{B_{i + w}^{2} + B_{i + w + 1}^{2}}\end{matrix} & {{{- R_{i}}R_{i + 1}} - {B_{i}B_{i + 1}}} & \ldots & {{{- R_{i}}R_{i + w}} - {B_{i}B_{i + w}}} & \begin{matrix}{{{- R_{i}}R_{i + w + 1}} -} \\{B_{i}B_{i + w + 1}}\end{matrix} & \; \\\; & \begin{matrix}{{{- R_{i}}R_{i + 1}} -} \\{B_{i}B_{i + 1}}\end{matrix} & \begin{matrix}{R_{i}^{2} + R_{i + w}^{2} +} \\{R_{i + w + 1}^{2} + B_{i}^{2} +} \\{B_{i + w}^{2} + B_{i + w + 1}^{2}}\end{matrix} & \ldots & \begin{matrix}{{{- R_{i + 1}}R_{i + w}} -} \\{B_{i + 1}B_{i + w}}\end{matrix} & \begin{matrix}{{{- R_{i + 1}}R_{i + w + 1}} -} \\{B_{i + 1}B_{i + w + 1}}\end{matrix} & \; \\\; & \vdots & \vdots & \ddots & \vdots & \vdots & \; \\\; & \begin{matrix}{{{- R_{i}}R_{i + w}} -} \\{B_{i}B_{i + w}}\end{matrix} & \begin{matrix}{{{- R_{i + 1}}R_{i + w}} -} \\{B_{i + 1}B_{i + w}}\end{matrix} & \ldots & \begin{matrix}{R_{i}^{2} + R_{i + 1}^{2} +} \\{R_{i + w + 1}^{2} + B_{i}^{2} +} \\{B_{i + 1}^{2} + B_{i + w + 1}^{2}}\end{matrix} & \begin{matrix}{{{- R_{i + w}}R_{i + w + 1}} -} \\{B_{i + w}B_{i + w + 1}}\end{matrix} & \; \\\; & \begin{matrix}{{{- R_{i}}R_{i + w + 1}} -} \\{B_{i}B_{i + w + 1}}\end{matrix} & \begin{matrix}{{{- R_{i + 1}}R_{i + w + 1}} -} \\{B_{i + 1}B_{i + w + 1}}\end{matrix} & \ldots & \begin{matrix}{{{- R_{i + w}}R_{i + w + 1}} -} \\{B_{i + w}B_{i + w + 1}}\end{matrix} & \begin{matrix}{R_{i}^{2} + R_{i + 1}^{2} +} \\{R_{i + w}^{2} + B_{i}^{2} +} \\{B_{i + 1}^{2} + B_{i + w}^{2}}\end{matrix} & \; \\\; & \vdots & \vdots & \ldots & \vdots & \vdots & \ddots\end{pmatrix}\begin{matrix}\begin{matrix}\begin{matrix}{{{i'}{th}\mspace{14mu} {row}}\;} \\\; \\\;\end{matrix} \\\; \\{{\left( {i + 1} \right)'}{th}\mspace{14mu} {row}} \\\;\end{matrix} \\\; \\\; \\{\; {{\left( {i + w} \right)'}{th}\mspace{14mu} {row}}} \\\; \\\; \\{{\left( {i + w + 1} \right)'}{th}\mspace{14mu} {row}}\end{matrix}}}{\begin{matrix}{\mspace{146mu} {{i'}{th}\mspace{14mu} {column}}} & {\mspace{25mu} {{\left( {i + 1} \right)'}{th}\mspace{14mu} {column}}} & {\mspace{34mu} {{\left( {i + w} \right)'}{th}{\mspace{11mu} \;}{column}}\mspace{14mu}}\end{matrix}{\left( {i + w + 1} \right)'}{th}\mspace{14mu} {col}}}} & \left\lbrack {{Expression}\mspace{14mu} 12} \right\rbrack\end{matrix}$

Q_(c) ^(T)Q_(c) can be calculated for the entire image by first makingall the elements of Q_(c) ^(T)Q_(c) zero and then while sequentiallyshifting the position of the top left pixel in the local area (namely,i), adding the local coefficient represented by expression 10 to theglobal coefficient matrix Q_(c) ^(T)Q_(c). In this process, the positionin the local area may be shifted by one pixel vertically andhorizontally so that the pixels overlap each other. Alternatively, theposition in the local area may be shifted by two pixels vertically andhorizontally so that the pixels are adjacent to each other withoutoverlapping each other. In the latter case, artifact may be generated atthe border of the local area, but substantially the same effect can beprovided as that in the former case by a smaller amount of calculation.

By the above-described procedure, H_(T), Q_(s), Q_(m) and Q_(c)^(T)Q_(c) can be calculated. H_(s) included in the fourth term ofexpression 3 is an operator for spatially decreasing the resolution ofG_(HH). This is used in order to impose the color correlation constraintdescribed above between the medium area of G_(HH) and R_(LH), B_(LH).

The value of each of the weights λ_(s), λ_(m) and λ_(c) to therespective constraint is set so as to improve the quality of the imageG_(HH) to be generated. For example, one criterion for setting thevalues of the weights λ_(s), λ_(m) and λ_(c) is whether, in expression3, the magnitudes of the terms (the four terms including the first term)weighted by these parameters substantially match one another at theorder level. Where the magnitudes of these four terms do not match atthe order level, the term having a large value is dominant as theconstraining condition and the term having a small value is noteffective as the constraining condition.

By solving the simultaneous expressions of expression 5 regarding theobserved image G_(HL) using the matrices and the weights calculated asdescribed above, G_(HH) can be generated.

By weighting as in expression 1 in accordance with the correlationbetween R and B in the local area represented by expression 12, G_(HH)which is visually more natural can be reproduced in accordance with thecorrelation between R and B.

$\begin{matrix}{\; {{{q_{i}^{\tau}q_{i}} = {{W\begin{pmatrix}\ddots & \; & \; & \; & \; & \; & \; \\\; & \begin{matrix}{R_{i + 1}^{2} + R_{i + w}^{2} +} \\{R_{i + w + 1}^{2} + B_{i + 1}^{2} +} \\{B_{i + w}^{2} + B_{i + w + 1}^{2}}\end{matrix} & {{{- R_{i}}R_{i + 1}} - {B_{i}B_{i + 1}}} & \ldots & {{{- R_{i}}R_{i + w}} - {B_{i}B_{i + w}}} & \begin{matrix}{{{- R_{i}}R_{i + w + 1}} -} \\{B_{i}B_{i + w + 1}}\end{matrix} & \; \\\; & \begin{matrix}{{{- R_{i}}R_{i + 1}} -} \\{B_{i}B_{i + 1}}\end{matrix} & \begin{matrix}{R_{i}^{2} + R_{i + w}^{2} +} \\{R_{i + w + 1}^{2} + B_{i}^{2} +} \\{B_{i + w}^{2} + B_{i + w + 1}^{2}}\end{matrix} & \ldots & \begin{matrix}{{{- R_{i + 1}}R_{i + w}} -} \\{B_{i + 1}B_{i + w}}\end{matrix} & \begin{matrix}{{{- R_{i + 1}}R_{i + w + 1}} -} \\{B_{i + 1}B_{i + w + 1}}\end{matrix} & \; \\\; & \vdots & \vdots & \ddots & \vdots & \vdots & \; \\\; & \begin{matrix}{{{- R_{i}}R_{i + w}} -} \\{B_{i}B_{i + w}}\end{matrix} & \begin{matrix}{{{- R_{i + 1}}R_{i + w}} -} \\{B_{i + 1}B_{i + w}}\end{matrix} & \ldots & \begin{matrix}{R_{i}^{2} + R_{i + 1}^{2} +} \\{R_{i + w + 1}^{2} + B_{i}^{2} +} \\{B_{i + 1}^{2} + B_{i + w + 1}^{2}}\end{matrix} & \begin{matrix}{{{- R_{i + w}}R_{i + w + 1}} -} \\{B_{i + w}B_{i + w + 1}}\end{matrix} & \; \\\; & \begin{matrix}{{{- R_{i}}R_{i + w + 1}} -} \\{B_{i}B_{i + w + 1}}\end{matrix} & \begin{matrix}{{{- R_{i + 1}}R_{i + w + 1}} -} \\{B_{i + 1}B_{i + w + 1}}\end{matrix} & \ldots & \begin{matrix}{{{- R_{i + w}}R_{i + w + 1}} -} \\{B_{i + w}B_{i + w + 1}}\end{matrix} & \begin{matrix}{R_{i}^{2} + R_{i + 1}^{2} +} \\{R_{i + w}^{2} + B_{i}^{2} +} \\{B_{i + 1}^{2} + B_{i + w}^{2}}\end{matrix} & \; \\\; & \vdots & \vdots & \ldots & \vdots & \vdots & \ddots\end{pmatrix}}\begin{matrix}\begin{matrix}\begin{matrix}{{{i'}{th}\mspace{14mu} {row}}\;} \\\; \\\;\end{matrix} \\\; \\{{\left( {i + 1} \right)'}{th}\mspace{14mu} {row}} \\\;\end{matrix} \\\; \\\; \\{\; {{\left( {i + w} \right)'}{th}\mspace{14mu} {row}}} \\\; \\\; \\{{\left( {i + w + 1} \right)'}{th}\mspace{14mu} {row}}\end{matrix}}}{\begin{matrix}{\mspace{146mu} {{i'}{th}\mspace{14mu} {column}}} & {\mspace{45mu} {{\left( {i + 1} \right)'}{th}\mspace{14mu} {column}}} & {\mspace{50mu} {{\left( {i + w} \right)'}{th}{\mspace{11mu} \;}{column}}\mspace{14mu}}\end{matrix}\mspace{11mu} {\left( {i + w + 1} \right)'}{th}\mspace{14mu} {col}}}} & \left\lbrack {{Expression}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Here, the distribution of weight W is set such that the weight W has avalue of 0 to 1 in accordance with the correlation value between R and Bin the local area (−1 to 1). For example, FIG. 6 shows the relationshipof the local correlation ρ between R and B against the weight W.Regarding the area of 2×2 pixels shown in FIG. 4, the local correlationρ between R and B is represented by expression 14.

$\begin{matrix}{\rho = \frac{\sum\limits_{i = 1}^{4}{\left( {R_{i} - \overset{\_}{R}} \right)\left( {B_{i} - \overset{\_}{B}} \right)}}{\sqrt{\sum\limits_{i = 1}^{4}\left( {R_{i} - \overset{\_}{R}} \right)^{2}}\sqrt{\sum\limits_{i = 1}^{4}\left( {B_{i} - \overset{\_}{B}} \right)^{2}}}} & \left\lbrack {{Expression}\mspace{14mu} 14} \right\rbrack\end{matrix}$

Once the local correlation ρ between R and B is obtained, the weight Wis determined based on the relationship shown in FIG. 6.

The local correlation between R and B does not need to be obtained foran area of 2×2 pixels, and may be obtained for a larger rectangular areaof 3 pixels×3 pixels, 4 pixels×4 pixels, 5 pixels×5 pixels or the like.The local correlation between R and B may be obtained for a circulararea or a polygonal area having four or more sides, or a weight whichputs importance on an area in the vicinity of the pixel of interest maybe used for calculation by the Gaussian function or the like. By usingsuch a calculation method, the calculation of the correlation value canbe made more isotropic for the pattern of the image.

In this embodiment, the constraint Q_(c) on the color correlation iscalculated using the R, G and B levels themselves. Substantially thesame effect can be provided by using the R, G and B gradients instead ofthe levels themselves. Such an example will be described below.

The following is known regarding a motion in a moving image: assumingthat the brightness is the same between corresponding points, therelationship of optical flow approximately holds regarding a motionvector (u, v) in the image. The optical flow of each of R, G and B isrepresented by expression 15.

$\begin{matrix}{{{{\frac{\partial R}{\partial x}u} + {\frac{\partial R}{\partial y}v} + \frac{\partial R}{\partial t}} = 0}{{{\frac{\partial G}{\partial x}u} + {\frac{\partial G}{\partial y}v} + \frac{\partial G}{\partial t}} = 0}{{{\frac{\partial B}{\partial x}u} + {\frac{\partial B}{\partial y}v} + \frac{\partial B}{\partial t}} = 0}} & \left\lbrack {{Expression}\mspace{14mu} 15} \right\rbrack\end{matrix}$

Here, assuming that the motion vector (u, v) in the image is the sameamong R, G and B, expression 16 is obtained.

$\begin{matrix}{{{{\frac{\partial R}{\partial x}\frac{\partial G}{\partial y}} - {\frac{\partial R}{\partial y}\frac{\partial G}{\partial x}}} = 0}{{{\frac{\partial R}{\partial y}\frac{\partial G}{\partial t}} - {\frac{\partial R}{\partial t}\frac{\partial G}{\partial y}}} = 0}{{{\frac{\partial R}{\partial t}\frac{\partial G}{\partial x}} - {\frac{\partial R}{\partial x}\frac{\partial G}{\partial t}}} = 0}{{{\frac{\partial B}{\partial x}\frac{\partial G}{\partial y}} - {\frac{\partial B}{\partial y}\frac{\partial G}{\partial x}}} = 0}{{{\frac{\partial B}{\partial y}\frac{\partial G}{\partial t}} - {\frac{\partial B}{\partial t}\frac{\partial G}{\partial y}}} = 0}{{{\frac{\partial B}{\partial t}\frac{\partial G}{\partial x}} - {\frac{\partial B}{\partial x}\frac{\partial G}{\partial t}}} = 0}} & \left\lbrack {{Expression}\mspace{14mu} 16} \right\rbrack\end{matrix}$

In expression 16, R and G are considered for the value of one pixel.However, ∂/∂x and ∂/∂y are represented as matrices by differenceexpansion. Hence, where R and G are considered as a vector, expression17 is obtained for the entirety of the moving image.

$\begin{matrix}{{{\left( {{\frac{\partial}{\partial x}R\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}R\frac{\partial}{\partial x}}} \right)G} = 0}{{\left( {{\frac{\partial}{\partial y}R\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}R\frac{\partial}{\partial y}}} \right)G} = 0}{{\left( {{\frac{\partial}{\partial t}R\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}R\frac{\partial}{\partial t}}} \right)G} = 0}{{\left( {{\frac{\partial}{\partial x}B\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}B\frac{\partial}{\partial x}}} \right)G} = 0}{{\left( {{\frac{\partial}{\partial y}B\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}B\frac{\partial}{\partial y}}} \right)G} = 0}{{\left( {{\frac{\partial}{\partial t}B\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}B\frac{\partial}{\partial t}}} \right)G} = 0}} & \left\lbrack {{Expression}\mspace{14mu} 17} \right\rbrack\end{matrix}$

In order to allow as many expressions as possible of expression 17 tohold at the same time for the entire image, G_(HH) which minimizesexpression 18 is obtained.

$\begin{matrix}{{{\left( {{\frac{\partial}{\partial x}R_{LH}\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}R_{LH}\frac{\partial}{\partial x}}} \right)H_{s}G_{HH}}}^{2} + {{\left( {{\frac{\partial}{\partial y}R_{LH}\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}R_{LH}\frac{\partial}{\partial y}}} \right)H_{s}G_{HH}}}^{2} + {{\left( {{\frac{\partial}{\partial t}R_{LH}\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}R_{LH}\frac{\partial}{\partial t}}} \right)H_{s}G_{HH}}}^{2} + {{\left( {{\frac{\partial}{\partial x}B_{LH}\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}B_{LH}\frac{\partial}{\partial x}}} \right)H_{s}G_{HH}}}^{2} + {{\left( {{\frac{\partial}{\partial y}B_{LH}\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}B_{LH}\frac{\partial}{\partial y}}} \right)H_{s}G_{HH}}}^{2} + {{\left( {{\frac{\partial}{\partial t}B_{LH}\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}B_{LH}\frac{\partial}{\partial t}}} \right)H_{s}G_{HH}}}^{2}} & \left\lbrack {{Expression}\mspace{14mu} 18} \right\rbrack\end{matrix}$

G_(HH) which minimizes expression 18 can be obtained by finding G_(HH)which makes zero the expression obtained by performing partialdifferentiation on expression 18 with G_(HH). Thus, expression 19 isobtained.

$\begin{matrix}{{\left( {{{\left( {{\frac{\partial}{\partial x}R_{LH}\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}R_{LH}\frac{\partial}{\partial x}}} \right)H_{s}}}^{2} + {{\left( {{\frac{\partial}{\partial y}R_{LH}\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}R_{LH}\frac{\partial}{\partial y}}} \right)H_{s}}}^{2} + {{\left( {{\frac{\partial}{\partial t}R_{LH}\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}R_{LH}\frac{\partial}{\partial t}}} \right)H_{s}}}^{2} + {{\left( {{\frac{\partial}{\partial x}B_{LH}\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}B_{LH}\frac{\partial}{\partial x}}} \right)H_{s}}}^{2} + {{\left( {{\frac{\partial}{\partial y}B_{LH}\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}B_{LH}\frac{\partial}{\partial y}}} \right)H_{s}}}^{2} + {{\left( {{\frac{\partial}{\partial t}B_{LH}\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}B_{LH}\frac{\partial}{\partial t}}} \right)H_{s}}}^{2}} \right)G_{HH}} = 0} & \left\lbrack {{Expression}\mspace{14mu} 19} \right\rbrack\end{matrix}$

By comparing expression 19 and the fourth term of expression 3,expression 20 is obtained.

$\begin{matrix}{Q_{c} = {\left( {{\frac{\partial}{\partial x}R_{LH}\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}R_{LH}\frac{\partial}{\partial x}}} \right) + \left( {{\frac{\partial}{\partial y}R_{LH}\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}R_{LH}\frac{\partial}{\partial y}}} \right) + \left( {{\frac{\partial}{\partial t}R_{LH}\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}R_{LH}\frac{\partial}{\partial t}}} \right) + \left( {{\frac{\partial}{\partial x}B_{LH}\frac{\partial}{\partial y}} - {\frac{\partial}{\partial y}B_{LH}\frac{\partial}{\partial x}}} \right) + \left( {{\frac{\partial}{\partial y}B_{LH}\frac{\partial}{\partial t}} - {\frac{\partial}{\partial t}B_{LH}\frac{\partial}{\partial y}}} \right) + \left( {{\frac{\partial}{\partial t}B_{LH}\frac{\partial}{\partial x}} - {\frac{\partial}{\partial x}B_{LH}\frac{\partial}{\partial t}}} \right)}} & \left\lbrack {{Expression}\mspace{14mu} 20} \right\rbrack\end{matrix}$

The local correlation is not limited to being considered regarding thespatial vicinity area as described above, but may be consideredregarding the temporal and spatial vicinity, needless to say. Forexample, a vicinity area as shown in FIG. 20 may be considered, in whichcase a more stable effect can be provided. FIG. 19 shows a group of R, Gand B pixels which form a color image of 2×2×2 pixels. Here, it isassumed that R, G and B are correlated to one another. Namely, it isassumed that the ratio among R, G and B is uniform in all the pixels.Thus, expression 21 holds.

$\begin{matrix}{{\frac{G_{1}}{R_{1}} = {\frac{G_{2}}{R_{2}} = {\frac{G_{3}}{R_{3}} = {\frac{G_{4}}{R_{4}} = {\frac{G_{5}}{R_{5}} = {\frac{G_{6}}{R_{6}} = {\frac{G_{7}}{R_{7}} = \frac{G_{8}}{R_{8}}}}}}}}}{\frac{G_{1}}{B_{1}} = {\frac{G_{2}}{B_{2}} = {\frac{G_{3}}{B_{3}} = {\frac{G_{4}}{B_{4}} = {\frac{G_{5}}{B_{5}} = {\frac{G_{6}}{B_{6}} = {\frac{G_{7}}{B_{7}} = \frac{G_{8}}{B_{8}}}}}}}}}} & \left\lbrack {{Expression}\mspace{14mu} 21} \right\rbrack\end{matrix}$

For selecting two out of each eight ratios, there are ₈C₂=28combinations. Thus, expressions 22 and 23 are obtained.

G ₁ R ₂ −G ₂ R ₁=0

G ₁ R ₃ −G ₃ R ₁=0

G ₁ R ₄ −G ₄ R ₁=0

G ₁ R ₅ −G ₅ R ₁=0

G ₁ R ₆ −G ₆ R ₁=0

G ₁ R ₇ −G ₇ R ₁=0

G ₁ R ₃ −G ₃ R ₁=0

G ₂ R ₃ −G ₃ R ₂=0

G ₂ R ₄ −G ₄ R ₂=0

G ₂ R ₅ −G ₅ R ₂=0

G ₂ R ₆ −G ₆ R ₂=0

G ₂ R ₇ −G ₇ R ₂=0

G ₂ R ₃ −G ₃ R ₂=0

G ₃ R ₄ −G ₄ R ₃=0

G ₃ R ₅ −G ₅ R ₃=0

G ₃ R ₆ −G ₆ R ₃=0

G ₃ R ₇ −G ₇ R ₃=0

G ₃ R ₃ −G ₃ R ₃=0

G ₄ R ₅ −G ₅ R ₄=0

G ₄ R ₆ −G ₆ R ₄=0

G ₄ R ₇ −G ₇ R ₄=0

G ₄ R ₃ −G ₃ R ₄=0

G ₅ R ₆ −G ₆ R ₅=0

G ₅ R ₇ −G ₇ R ₅=0

G ₅ R ₃ −G ₃ R ₅=0

G ₆ R ₇ −G ₇ R ₆=0

G ₆ R ₃ −G ₃ R ₆=0

G ₇ R ₃ −G ₃ R ₇=0  [Expression 22]

G ₁ B ₂ −G ₂ B ₁=0

G ₁ B ₃ −G ₃ B ₁=0

G ₁ B ₄ −G ₄ B ₁=0

G ₁ B ₅ −G ₅ B ₁=0

G ₁ B ₆ −G ₆ B ₁=0

G ₁ B ₇ −G ₇ B ₁=0

G ₁ B ₃ −G ₃ B ₁=0

G ₂ B ₃ −G ₃ B ₂=0

G ₂ B ₄ −G ₄ B ₂=0

G ₂ B ₅ −G ₅ B ₂=0

G ₂ B ₆ −G ₆ B ₂=0

G ₂ B ₇ −G ₇ B ₂=0

G ₂ B ₃ −G ₃ B ₂=0

G ₃ B ₄ −G ₄ B ₃=0

G ₃ B ₅ −G ₅ B ₃=0

G ₃ B ₆ −G ₆ B ₃=0

G ₃ B ₇ −G ₇ B ₃=0

G ₃ B ₃ −G ₃ B ₃=0

G ₄ B ₅ −G ₅ B ₄=0

G ₄ B ₆ −G ₆ B ₄=0

G ₄ B ₇ −G ₇ B ₄=0

G ₄ B ₃ −G ₃ B ₄=0

G ₅ B ₆ −G ₆ B ₅=0

G ₅ B ₇ −G ₇ B ₅=0

G ₅ B ₃ −G ₃ B ₅=0

G ₆ B ₇ −G ₇ B ₆=0

G ₆ B ₃ −G ₃ B ₆=0

G ₇ B ₃ −G ₃ B ₇=0  [Expression 23]

Expressions 22 and 23 are represented by matrices and vectors asexpressions 24 and 25.

$\begin{matrix}\begin{matrix}{{\begin{pmatrix}R_{2} & R_{1} & 0 & 0 & 0 & 0 & 0 & 0 \\R_{3} & 0 & R_{1} & 0 & 0 & 0 & 0 & 0 \\R_{4} & 0 & 0 & R_{1} & 0 & 0 & 0 & 0 \\R_{5} & 0 & 0 & 0 & R_{1} & 0 & 0 & 0 \\R_{6} & 0 & 0 & 0 & 0 & R_{1} & 0 & 0 \\R_{7} & 0 & 0 & 0 & 0 & 0 & R_{1} & 0 \\R_{8} & 0 & 0 & 0 & 0 & 0 & 0 & R_{1} \\0 & R_{3} & R_{2} & 0 & 0 & 0 & 0 & 0 \\0 & R_{4} & 0 & R_{2} & 0 & 0 & 0 & 0 \\0 & R_{5} & 0 & 0 & R_{2} & 0 & 0 & 0 \\0 & R_{6} & 0 & 0 & 0 & R_{2} & 0 & 0 \\0 & R_{7} & 0 & 0 & 0 & 0 & R_{2} & 0 \\0 & R_{8} & 0 & 0 & 0 & 0 & 0 & R_{2} \\0 & 0 & R_{4} & R_{3} & 0 & 0 & 0 & 0 \\0 & 0 & R_{5} & 0 & R_{3} & 0 & 0 & 0 \\0 & 0 & R_{6} & 0 & 0 & R_{3} & 0 & 0 \\0 & 0 & R_{7} & 0 & 0 & 0 & R_{3} & 0 \\0 & 0 & R_{8} & 0 & 0 & 0 & 0 & R_{3} \\0 & 0 & 0 & R_{5} & R_{4} & 0 & 0 & 0 \\0 & 0 & 0 & R_{6} & 0 & R_{4} & 0 & 0 \\0 & 0 & 0 & R_{7} & 0 & 0 & R_{4} & 0 \\0 & 0 & 0 & R_{8} & 0 & 0 & 0 & R_{4} \\0 & 0 & 0 & 0 & R_{6} & R_{5} & 0 & 0 \\0 & 0 & 0 & 0 & R_{7} & 0 & R_{5} & 0 \\0 & 0 & 0 & 0 & R_{8} & 0 & 0 & R_{5} \\0 & 0 & 0 & 0 & 0 & R_{7} & R_{6} & 0 \\0 & 0 & 0 & 0 & 0 & R_{8} & 0 & R_{6} \\0 & 0 & 0 & 0 & 0 & 0 & R_{8} & R_{7}\end{pmatrix}\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}} = {q_{R}\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}}} \\{= 0}\end{matrix} & \left\lbrack {{Expression}\mspace{14mu} 24} \right\rbrack \\\begin{matrix}{{\begin{pmatrix}B_{2} & B_{1} & 0 & 0 & 0 & 0 & 0 & 0 \\B_{3} & 0 & B_{1} & 0 & 0 & 0 & 0 & 0 \\B_{4} & 0 & 0 & B_{1} & 0 & 0 & 0 & 0 \\B_{5} & 0 & 0 & 0 & B_{1} & 0 & 0 & 0 \\B_{6} & 0 & 0 & 0 & 0 & B_{1} & 0 & 0 \\B_{7} & 0 & 0 & 0 & 0 & 0 & B_{1} & 0 \\B_{8} & 0 & 0 & 0 & 0 & 0 & 0 & B_{1} \\0 & B_{3} & B_{2} & 0 & 0 & 0 & 0 & 0 \\0 & B_{4} & 0 & B_{2} & 0 & 0 & 0 & 0 \\0 & B_{5} & 0 & 0 & B_{2} & 0 & 0 & 0 \\0 & B_{6} & 0 & 0 & 0 & B_{2} & 0 & 0 \\0 & B_{7} & 0 & 0 & 0 & 0 & B_{2} & 0 \\0 & B_{8} & 0 & 0 & 0 & 0 & 0 & B_{2} \\0 & 0 & B_{4} & B_{3} & 0 & 0 & 0 & 0 \\0 & 0 & B_{5} & 0 & B_{3} & 0 & 0 & 0 \\0 & 0 & B_{6} & 0 & 0 & B_{3} & 0 & 0 \\0 & 0 & B_{7} & 0 & 0 & 0 & B_{3} & 0 \\0 & 0 & B_{8} & 0 & 0 & 0 & 0 & B_{3} \\0 & 0 & 0 & B_{5} & B_{4} & 0 & 0 & 0 \\0 & 0 & 0 & B_{6} & 0 & B_{4} & 0 & 0 \\0 & 0 & 0 & B_{7} & 0 & 0 & B_{4} & 0 \\0 & 0 & 0 & B_{8} & 0 & 0 & 0 & B_{4} \\0 & 0 & 0 & 0 & B_{6} & B_{5} & 0 & 0 \\0 & 0 & 0 & 0 & B_{7} & 0 & B_{5} & 0 \\0 & 0 & 0 & 0 & B_{8} & 0 & 0 & B_{5} \\0 & 0 & 0 & 0 & 0 & B_{7} & B_{6} & 0 \\0 & 0 & 0 & 0 & 0 & B_{8} & 0 & B_{6} \\0 & 0 & 0 & 0 & 0 & 0 & B_{8} & B_{7}\end{pmatrix}\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}} = {q_{B}\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}}} \\{= 0}\end{matrix} & \left\lbrack {{Expression}\mspace{14mu} 25} \right\rbrack\end{matrix}$

From expressions 24 and 25, the following relationship is obtained.

$\begin{matrix}{{q\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}} = {{\begin{pmatrix}q_{R} \\q_{B}\end{pmatrix}\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}} = 0}} & \left\lbrack {{Expression}\mspace{14mu} 26} \right\rbrack\end{matrix}$

In the case where the area of 2×2×2 pixels is considered as a localarea, the correlation among R, G and B can be evaluated by the norm ofthe left side of expression 25 (i.e., expression 27).

$\begin{matrix}{\mspace{560mu} \left\lbrack {{Expression}\mspace{14mu} 27} \right\rbrack} \\{{{q\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}}}^{2} = {\begin{pmatrix}G_{1} & G_{2} & G_{3} & G_{4} & G_{5} & G_{6} & G_{7} & G_{8}\end{pmatrix}q^{T}{q\begin{pmatrix}G_{1} \\G_{2} \\G_{3} \\G_{4} \\G_{5} \\G_{6} \\G_{7} \\G_{8}\end{pmatrix}}}}\end{matrix}$

In the case where the area of 2×2×2 pixels is considered as a local areain an image larger than 2×2×2 pixels, the local relationship issequentially added to the global relational expression like in the casewhere the spatial vicinity area of 2×2 pixels is considered above. As aresult, the global relational expression can be obtained.

Instead of considering the local correlation regarding the vicinity areaof 2×2×2 pixels, local correlation may be considered regarding only fourpixels as shown in FIG. 21, which include the pixel of interest andthree pixels adjacent thereto temporally and spatially. In this case,substantially the same effect can be provided with a smaller amount ofcalculation than considering the local correlation regarding thevicinity area of 2×2×2 pixels.

In the case where an optical flow (motion vector field) is accuratelyobtained by assuming a local correlation in a direction of the opticalflow, i.e., in the direction of the motion vector instead of byconsidering the local correlation regarding the vicinity area in thetemporal direction as described above, a more stable effect can beprovided. Moreover, where the motion vector is obtained with a sub pixelprecision, the motion information of the sub pixel precision can beeffectively used by performing weighting using a value of the motionvector which is lower than the decimal point.

Regarding expression 3 described above, the constraining conditions ofthe first through fourth terms of the right side do not need to be usedat the same time. The following modifications may be used. Namely, onlythe first and second terms of expression 1 may be used as:

J=(H _(T) G _(HH) −G _(HL))²+λ_(s)(Q _(s) G _(HH))²  [Expression 28]

Alternatively, expression 29 based on only the first and fourth terms ofexpression 3 may be used.

J=(H _(T) G _(HH) −G _(HL))²+λ_(c)(Q _(c) H _(s) G _(HH))²  [Expression29]

Alternatively, expression 30 based on only the first, second and thirdterms of expression 3 may be used.

J=(H _(T) G _(HH) −G _(HL))²+λ_(s)(Q _(s) G _(HH))²+λ_(m)(Q _(m) G_(HH))²  [Expression 30]

Alternatively, expression 31 based on only the first, second and fourthterms of expression 3 may be used.

J=(H _(T) G _(HH) −G _(HL))²+λ_(s)(Q _(s) G _(HH))²+λ_(c)(Q _(c) H _(s)G _(HH))²  [Expression 31]

Using any of these modifications, the resolution of the G component canbe increased with a smaller amount of calculation than performingcalculations by expression 1 using all the constraining conditions ofall the four terms of expression 3. Note, however, that use of any ofthese modifications means alleviating the constraining conditions and sothe level of the resolution of the generated image may be sensedslightly lower.

For a scene for which motion detection is difficult, the processing ofincreasing the resolution is performed using an expression which doesnot use the result of motion detection (expression 28, 29 or 31). Thus,the generation of artifact (disturbance of image, noise) caused to thequality of the output image by a wrong detection of motion can besuppressed. Whether the motion detection from a scene is difficult ornot can be determined based on (a) the difference in the results ofmotion detections performed in temporal bidirections or (b) the minimumvalue in the search range of the evaluation value at the time of motiondetection represented by expression 1 or 2.

In the former case (a), assuming that the motion detection result in theforward temporal direction at (x, y) in the image in the baseline frameis (u, v), the determination is made as follows. After the forwarddirection motion detection, motion detection is performed in the reversedirection based on the reference frame at the time of the forwarddirection motion detection. When the motion detection result at (x+u,y+v) is (−u, −v), the motion detection results in both directions areconsistent and reliable. By contrast, when the motion detection resultis not (−u, −v), for example, when the motion detection result isdifferent from (−u, −v) by a certain threshold value or greater, it canbe determined that the motion detection is difficult.

Similarly in the latter case (b), when the difference from the minimumvalue in the search range of the evaluation value at the time of motiondetection performed using SSD or SAD is, for example, a predeterminedthreshold value or greater, it can be determined that the motiondetection is difficult.

For a scene with no color correlation, expression 28 or 30 is usable.When the weighting method represented by expression 13 is used for ascene with no color correlation, the weight is decreased. Therefore, insuch a case, expression 30 is automatically used with no specificoperation.

FIG. 1 will be referred to again. The upconverter 107 for R and B usesthe moving image of the G component having the resolution thereofincreased by the upconverter 106 for G to increase the resolutions ofthe moving images of R and B components taken by the first imagingsection 103 with the imaging conditions of a short-time exposure, a lowresolution and a high frame rate.

Hereinafter, with reference to FIG. 7, processing of increasing theresolutions of the moving images of R and B components will be describedin detail.

FIG. 7 shows the structure of the upconverter 107 for R and B in moredetail. In FIG. 7, elements common to those in the imaging andprocessing device shown in FIG. 1 bear identical reference numeralsthereto and descriptions thereof will be omitted.

The upconverter 107 for R and B includes a downconverter 110, acoefficient estimation section 111 and a restoration filter 112.

The downconverter 110 spatially decreases the resolution of the Gcomponent (G_(HH)) having the resolution thereof increased by theupconverter 106 for G and outputs G_(LH). The coefficient estimationsection 111 estimates a filter coefficient of the restoration filter 112(restoration filter H+) from G_(HH) and G_(LH). As the restorationfilter 112, a known filter such as a Wiener filter, a general reversefilter or the like is usable. Such a restoration filter estimates a highresolution side signal from a low resolution signal, using therelationship between G_(HH) and G_(LH). The restoration filter 112restores R_(HH) and B_(HH) from R_(LH) and B_(LH), using the filtercoefficient estimated by the coefficient estimation section 111.

The processing performed by the upconverter 107 for R and G forincreasing the resolutions of R and B is not limited to theabove-described, so-called reconstruction type processing, and may beany other type of resolution increasing processing. The resolutionincreasing processing in another example will be described, hereinafter.

FIG. 8 shows a structure of an upconverter 107 for R and G in anotherexample in detail. Here, the upconverter 107 for R and G increases theresolutions of the R component and the B component by superimposing ahigh frequency component of G on the R component and the B component,which have been processed with interpolation based expansion, inaccordance with the correlation between R and B.

The upconverter 107 for R and G shown in FIG. 8 includes a downconverter113, an interpolation based expansion section 114, a local correlationcalculation section 115, a weight generation section 116, a counter 117,an image memory 118, and a normalization section 119.

The upconverter 107 for R and G performs processing in units of frameswhich form a moving image. First, at the beginning of each frame, thecontent of the counter 117 and the content of the image memory 118 arecleared by filling the counter 117 and the image memory 118 with zeros.

The downconverter 113 spatially decreases the resolution of the Gcomponent (G_(HH)) having the resolution thereof increased by theupconverter 106 for G and outputs G_(LH).

The interpolation based expansion section 114 receives R_(LH) and B_(LH)taken with a low resolution and a high frame rate and the imagingconditions, and performs interpolation based expansion on R_(LH) andB_(LH) such that R_(LH) and B_(LH) have the same number of pixels asthat of G_(HH).

The local correlation calculation section 115 calculates a localcorrelation value between R_(LH) and B_(LH) in a local area of about 2×2pixels or 3×3 pixels. For calculating the local correlation value of anarea of 2×2 pixels, the local correlation calculation section 115 canuse, for example, expression 14.

The weight generation section 116 generates a weight in accordance withthe correlation value calculated by the local correlation calculationsection 115. FIG. 9 shows an example of the relationship between thecorrelation value ρ and weight W2. The weight generation section 116finds the weight W2 based on the correlation value ρ and therelationship shown in FIG. 9.

As shown in FIG. 8, the weight generated and output by the weightgeneration section 116 is multiplied by a difference between G_(HH) andG_(LH) (namely, the high frequency component of G) to update the imagememory 118. In more detail, after the multiplication, an address inaccordance with the position in the image memory 118 at which the imagedata is stored is specified. The multiplication result and the valueheld at the address are added together, and the value of the address isrewritten by the addition result.

In this process, the target to be written in the image memory 118 may beone pixel or pixels in a range obtained by calculating the localcorrelation value. Note that when a high frequency component issuperimposed on a plurality of pixels as in the latter case, the highfrequency component may be superimposed on the same pixel a plurality oftimes, depending on the manner of setting of the area for which thelocal correlation is calculated (namely, the manner of incrementing inthe image). In consideration of such a case, the upconverter 107 for Rand B shown in FIG. 8 uses the counter 117. The counter 117 stores thenumber of times the high frequency component is superimposed for eachpixel.

The normalization section 119 divides the high frequency componentsuperimposed a plurality of times by the number of times of write whichis stored in the counter 117 for each pixel. The normalized highfrequency component is superimposed on the R and B images processed withinterpolation and expansion by the interpolation and expansion section114 and output as R_(HH) and B_(HH).

By increasing the resolutions of R and B by the above-described method,the resolutions of R and B can be increased while the local colorbalance is maintained. As a result, the resolutions can be increasedwhile the generation of false colors is suppressed.

FIG. 9 shows the linear relationship as an example of the relationshipbetween the correlation value and the weight. This is merely an example,and the relationship may be made nonlinear in consideration of the ycharacteristic at the time of imaging or display. Alternatively, theweight may be normalized by (local average of R)/(local average of G)for R, and by (local average of B)/(local average of G) for B. Suchnormalization can adjust the amplitude of the high frequency componentof G, which is superimposed on R and B, in accordance with the pixelvalues of R, G and B. Thus, the sense of discomfort caused at the timeof observation by excessive superimposition of the high frequency can bereduced. As the local averages of R, G and B, the pixel values of R_(LH)and B_(LH) processed with interpolation and expansion, and the pixelvalue of G_(LH) obtained by downconverting G_(HH) by the downconverter113 shown in FIG. 8 may be used.

In the case where the high frequency component is superimposed on the Rand B images processed with interpolation and expansion only once foreach pixel, the image memory 118, the counter 117 and the normalizationsection 119 shown in FIG. 8 are unnecessary, and the structure shown inFIG. 10 can be adopted. FIG. 10 shows a modification of the structure ofthe upconverter 107 for R and B. As shown in FIG. 10, the upconverter107 for R and B can be realized with a simpler structure than that ofFIG. 8 and substantially the same effect can be provided.

The resolution increasing processing for R and B performed by theupconverter 107 for R and B is not limited to the above-described,so-called reconstruction type super resolution processing or theprocessing of superimposing the high frequency component of G on the Rand B components. The relationship between the G component having theresolution thereof increased and the G component obtained by decreasingthe resolution thereof (i.e., G_(HH) and G_(LH)) may be learned and theresolutions of the R and B components may be increased based on theresult of learning.

The learning is not limited to being performed during the processing ofthe input image, and may be performed with a learning pattern preparedin advance. In this case, the relationship between the G componenthaving a low resolution and the G component having a high resolution, aswell as the relationship between the RGB components having a lowresolution and the RGB components having a high resolution, can belearned.

In the above description, an output signal is a component of each colorof R, G and B. Hereinafter, an imaging and processing device whichoutputs each output signal of R, G and B after converting such a signalinto a luminance signal and a color difference signal will be described.

FIG. 11 shows a structure of an imaging and processing device 11 in amodification of this embodiment. In FIG. 11, elements common to those inthe above-described imaging and processing devices bear identicalreference numerals thereto and descriptions thereof will be omitted.

The image processing section 105 includes a color difference calculationsection 129 and a luminance calculation section 130 in addition to theupconverter 106 for G, the interpolation and expansion section 114 andthe downconverter 113.

The control section 120 receives a signal of an R component and a Bcomponent processed with interpolation and expansion by theinterpolation and expansion section 114 and a signal of a G componenthaving the resolution thereof decreased by the downconverter 113,converts the signals into color difference signals (Cb signal, Crsignal) by the calculation of expression 32 and outputs the resultantsignals.

C _(b)=0.564(B−Y)=−0.169R−0.331G+0.500B

Cr=0.713(R−Y)=0.500R−0.419G−0.081B  [Expression 32]

The luminance calculation section 130 receives the signal of the Rcomponent and the B component processed with interpolation and expansionby the interpolation and expansion section 114 and the signal of the Gcomponent having the resolution thereof increased by the upconverter forG, converts the signals into a luminance signal (Y signal) by thecalculation of expression 33 and outputs the resultant signal.

Y=0.299R+0.587G+0.114B  [Expression 33]

As can be understood from the explanation on the color differencecalculation section 129 and the luminance calculation section 130, Ghaving the resolution thereof decreased is used for calculating thecolor difference components Cb and Cr, whereas G having the resolutionthereof increased is used for calculating the luminance component Y.Thus, the resolution of the output image can be increased while thegeneration of false colors is suppressed.

In a stage after the image processing section 105, a block forconverting the Y, Cb and Cr signals into RGB signals may be furtherprovided so as to output signals of RGB components.

In the case where the output signal does not need to represent a colorimage and may represent a monochrome image, it is not necessary toincrease the resolution of the R component or the B component inconsideration of the visual sight. In this case, a structure shown inFIG. 12 may be adopted.

FIG. 12 shows a structure of an imaging and processing device 12according to a modification of this embodiment. Since it is notnecessary to increase the resolution of the R component or the Bcomponent, the imaging and processing device 12 does not include theupconverter 107 for R and B which is included in the imaging andprocessing device 1 (FIG. 1). The imaging and processing device 12outputs only the G_(HH) signal for green having the resolution thereofincreased.

The imaging and processing devices in this embodiment and modificationthereof image the G component with a high resolution, a long-timeexposure and a low frame rate and image the R component and the Bcomponent with a low resolution, a short-time exposure and a high framerate. This is merely an example. Regarding which color component(wavelength) is to be imaged with a high resolution, a long-timeexposure and a low frame rate, other examples may be adopted.

When it is well expected in advance that the B component appearsstrongly in a scene, for example, when the scene in water such as thesea, pool or the like is to be imaged, the B component may be imagedwith a high resolution, a long-time exposure and a low frame ratewhereas the R component and the B component may be imaged with a lowresolution, a short-time exposure and a high frame rate. In this way, animage which gives the observer a stronger impression that the resolutionis high can be presented.

For example, FIG. 13 shows a structure of an imaging and processingdevice 13 obtained as a result of generalizing the imaging andprocessing device 1. In FIG. 13, elements common to those in the imagingand processing device shown in FIG. 1 bear identical reference numeralsthereto and descriptions thereof will be omitted.

The imaging and processing device 13 includes an R component imagingsection 131, a G component imaging section 132, a B component imagingsection 133, a control section 134, a switching section 135, anupconverter 136 for HL, an upconverter 137 for LH, and an output section138. Hereinafter, a function of each element will be described togetherwith an operation of the imaging and processing device 13.

Visible light transmitted through the optical system 101 is divided interms of wavelength by a dichroic prism and imaged by the R componentimager 131, the G component imager 132 and the B component imager 133.The number of pixels of each of RGB components readable by the imagingsections 131, 132 and 133 can be independently and dynamically set bythe binning read method. The “binning read method” is a method of addingand reading charges accumulated in adjacent pixels. In the imagingsections 131, 132 and 133, the exposure time and frame rate can also beset similarly. The conditions for read are set by the control section134.

The control section 134 sets either one of the R component imager 131,the G component imager 132 and the B component imager 133 to performimaging with a high resolution, a long-time exposure and a low framerate (corresponding to G in Embodiment 1), and sets the remaining two toperform imaging with a low resolution, a short-time exposure and a highframe rate (corresponding to R and B in Embodiment 1), in accordancewith the distribution of color components in the scene.

At the start of the imaging, the distribution of the color components inthe scene is not known yet. Therefore, for example, G may be set to beimaged with a high resolution, a long-time exposure and a low framerate.

The switching section 135 performs a switching operation in accordancewith the setting of the imaging sections 131, 132 and 133 for the RGBcomponents provided by the control section, such that imaging data ofthe component set to be imaged with a high resolution, a long-timeexposure and a low frame rate is input to the upconverter 136 for HL andthe data of the other components is input to the upconverter 137 for LH.

The upconverter 136 for HL spatially increases the resolution of amoving image of the component taken with a high resolution, a long-timeexposure and a low frame rate by the same processing as that of theupconverter 106 for G (shown in, for example, FIG. 1).

The upconverter 137 for LH receives moving images of two systems (twocolor components) taken with a low resolution, a short-time exposure anda high frame rate and the moving image having the resolution thereofincreased by the upconverter 136 for HL and spatially increases theresolutions of the moving images of the two systems by the sameprocessing as that of the upconverter 107 for R and B (shown in, forexample, FIG. 1).

The output section 138 receives the moving images having the resolutionsthereof increased by the upconverter 136 for HL and the upconverter 137for LH and outputs the moving images of three systems of RGB inaccordance with the setting by the control section 134. Needless to say,the output section 138 may output the moving images, which are convertedinto images of another signal format such as, for example, a luminancesignal (Y) and color difference signals (Cb, Cr).

Embodiment 2

In Embodiment 1 described above, the imaging processing and theresolution increasing processing are performed by the same system.However, these two types of processing do not need to be performed bythe same system.

In this embodiment, the imaging processing and the resolution increasingprocessing are performed by different systems.

FIG. 14 shows an example of an image processing system in thisembodiment including an imaging device 901, a network 902 and aprocessing device 903. An image processing system according to thisembodiment can be constructed using a medium 906 instead of the network902. In FIG. 14, elements common to those in the imaging and processingdevice according to Embodiment 1 (shown in, for example, FIG. 1) bearidentical reference numerals thereto and descriptions thereof will beomitted.

The imaging device 901 includes a lens system 101, a dichroic mirror102, a first imaging section 103, a second imaging section 104, and animaging mode setting section 904.

The first imaging section 103 images an R component and a B component ofa color image with a short-time exposure, a low resolution and a highframe rate and outputs an R image R_(LH) and a B image B_(LH). Thesecond imaging section 104 images a G component of the color image witha long-time exposure, a high resolution and a low frame rate and outputsa G image G_(HL).

The imaging mode setting section 904 sets variable imaging conditionsof, for example, the frame rate, the exposure time and the like of thesecond imaging section 104, and writes information indicating the setconditions in a comment area in a header of a video signal and outputsthe information to the network 902 via an output section 905, or outputsthe information to the network 902 via the output section 905 asseparate data.

The output section 905 outputs the G image G_(HL), the R image R_(LH)and the B image B_(LH) taken by the imaging device 901, and also theinformation on the imaging conditions to the network 902 or the medium906.

The processing device 903 includes an image processing section 105. Theimage processing section 105 receives G_(HL), R_(LH) and B_(LH) and theinformation on the imaging conditions via the network 902 or the medium906, and outputs G_(HH), R_(HH) and B_(HH) each having the resolutionthereof increased spatially or temporally by the processing described inEmbodiment 1.

Owing to the above-described structure, the imaging device and theprocessing device, even though being of separate bodies and spatiallydiscrete from each other, can send and receive the moving image signalsand information on the imaging conditions via the network 902 or themedium 906. Thus, the processing device can output a moving image havinghigh temporal and spatial resolutions.

The network 902 may be a LAN (Local Area Network) constructed in a houseor a WAN (Wide Area Network) such as the Internet or the like.Alternatively, the network 902 may be a communication line of a USBstandard or an IEEE 1394 standard, and may be wireless or wired. Themedium 906 may be, for example, an optical disc, a removable disc suchas a removable hard disc or the like, or a flash memory card.

In the above embodiments, the imaging and processing device is describedas including any of the various structures shown in the figures. Forexample, the image processing section included in each structure isdescribed as a functional block. Such functional blocks may be realized,as hardware, as one semiconductor chip such as a digital signalprocessor (DSP) or the like or an IC, or may be realized using, forexample, a computer and software (computer program).

For example, FIG. 15 shows hardware of an image processing deviceconstructed by a computer.

The functional blocks of the image processing device in each embodimentand the hardware shown in FIG. 15 have the following correspondence. Inthe following, the correspondence will be described with the imageprocessing device 1 mainly shown in FIG. 1 as an example.

The lens system 101, the dichroic mirror 102, the first imaging section103 and the second imaging section 104 of the imaging and processingdevice 1 correspond to a camera 151 and an A/D converter 152 shown inFIG. 15. A temporary buffer (not shown) used by the image processingsection 105 in actual processing and the medium 906 correspond to aframe memory 153 or a hard disc drive (HDD) 160 shown in FIG. 15. Thecontrol section 120 and the image processing section 105 are realized bya CPU 154 shown in FIG. 15 for executing a computer program.

The computer program for causing the computer shown in FIG. 15 tooperate is stored on, for example, a ROM 155. Alternatively, thecomputer program may be stored on an optical disc or a magnetic disc.Still alternatively, the computer program may be transferred via a wiredor wireless network, broadcast or the like and stored on a RAM 156 ofthe computer.

The computer program is read onto the RAM 156 by the CPU 154 as aprocessor and extended. The CPU 154 executes coded instructions, whichform the substance of the computer program. A digital image signalobtained as a result of execution of each instruction is sent to, andtemporarily stored in, the frame memory 157, converted into an analogsignal by the D/A converter 158 and sent to, and displayed by, a display159.

The processing of the computer program for realizing the imageprocessing section 105 is described, for example, along with a flowchartexplained below.

For example, FIG. 16 is a flowchart showing a procedure of theprocessing according to the present invention. This processing will beexplained as the processing according to Embodiment 1, but may beconsidered as being executed independently by the imaging device 901 andthe processing device 903 in Embodiment 2.

First, in step S101, the first imaging section 103 and the secondimaging section 104 take a G image G_(HL) with a long-time exposure, ahigh resolution and a low frame rate and an R image R_(LH) and a B imageB_(LH) with a short-time exposure, a low resolution and a high framerate. In step S102, the upconverter 106 for G of the image processingsection 105 increases the resolution of the moving image of the Gcomponent. More specifically, this step can be divided into step S104and step S105. In step S104, the motion detection section 108 of theupconverter 106 for G performs motion detection. In step S105, thetemporal upconverter 109 uses the result of the motion detection or thelike to find G_(HH) which minimizes expression 3 based on expression 4.

Next in step S103, the upconverter 107 for R and B increases theresolution of the moving image of each of R and B components. Then, thecontrol section 120 determines whether or not the imaging has beencompleted. When it is determined that the imaging has not beencompleted, the processing is repeated from step S101. When it isdetermined that the imaging has been completed, the processing isterminated.

FIG. 17 is a flowchart showing a detailed procedure of step S103 shownin FIG. 16. This processing corresponds to the processing performed bythe upconverter 107 for R and B shown in FIG. 7.

In step S106 shown in FIG. 17, the downconverter 110 decreases theresolution of G_(HH) based on the imaging conditions. In step S107, thecoefficient estimation section 111 estimates a coefficient to be appliedto the restoration filter 112. In step S108, the estimated coefficientis applied to the restoration filter 112, and the restoration filter 112increases the resolutions of R_(LH) and B_(LH) and outputs R_(HH) and B.

FIG. 18 is a flowchart showing a detailed procedure of step S103 inanother example. This processing corresponds to the processing performedby the upconverter 107 for R and B shown in FIG. 8.

In step S109, the downconverter 113 decreases the resolution of G_(HH)based on the imaging conditions. In step S110, G_(LH) is subtracted fromG_(HH).

In the meantime, in step S111, the interpolation and expansion section114 performs interpolation and expansion on R_(LH) and B_(LH) based onthe imaging conditions. Based on the resultant signals, in step S112,the local correlation calculation section 115 calculates a localcorrelation value.

In step S113, the weight generation section 116 generates a weight. Instep S114, the counter 117 stores the number of times the high frequencycomponent is superimposed for each pixel. In step S115, the weightgenerated and output by the weight generation section 116 is multipliedby the difference between G_(HH) and G_(LH) (namely, the high frequencycomponent of G). In step S116, the image memory 118 is updated.

In step S117, the normalization section 119 divides the high frequencycomponent stored in the image memory 118 and superimposed a plurality oftimes by the number of times of write stored in the counter 117 for eachpixel to normalize the high frequency component.

In step S118, the normalized high frequency component is superimposed onthe R and B images processed with interpolation and expansion by theinterpolation and expansion section 114 and output as R_(HH) and B_(HH).

Various embodiments of the present invention have been described. InEmbodiments 1 and 2, the three components of R, G and B are separated bya dichroic mirror. The color separation is not limited to such a form.For example, a single imaging device for taking an image in threeseparate layers in the depth direction of R+G+B, R+B and R sequentiallymay be used. In this case also, an image of R+G or R+G+B is taken with ahigh resolution, a long-time exposure and a low frame rate, and theother images are taken with a low resolution, a short-time exposure anda high frame rate. These images are received and processed, and thussubstantially the same effect can be provided.

INDUSTRIAL APPLICABILITY

An imaging device and a processing device according to the presentinvention are useful for taking a high precision image by a camerahaving a reduced-sized imaging element and for a reproduction apparatusand a system for processing the resultant image. The imaging device andthe processing device according to the present invention can be realizedas a computer program.

1. An imaging and processing device, comprising: a separation sectionfor separating visible light into at least a first color component and asecond color component; a first imaging section for taking a movingimage of the first color component, the first imaging section takingimages which form the moving image with a first spatial resolution and afirst temporal resolution by exposure for a first charge accumulationtime period; a second imaging section for taking a moving image of thesecond color component, the second imaging section taking images whichform the moving image with a second spatial resolution higher than thefirst spatial resolution and a second temporal resolution lower than thefirst temporal resolution by exposure for a second charge accumulationtime period longer than the first charge accumulation time period; acontrol section for controlling imaging conditions of the first imagingsection and the second imaging section; and a processing section forgenerating a moving image of the second component having the temporaland spatial resolutions thereof increased, based on information on themoving image of the first color component and information on the movingimage of the second color component.
 2. An imaging and processingdevice, comprising: a separation section for separating visible lightinto at least a first color component and a second color component; afirst imaging section for taking a moving image of the first colorcomponent, the first imaging section taking images which form the movingimage with a first spatial resolution and a first temporal resolution byexposure for a first charge accumulation time period; a second imagingsection for taking a moving image of the second color component, thesecond imaging section taking images which form the moving image with asecond spatial resolution higher than the first spatial resolution and asecond temporal resolution lower than the first temporal resolution byexposure for a second charge accumulation time period longer than thefirst charge accumulation time period; a motion detection section forgenerating motion information based on a time-wise change of the movingimage of the first color component; and a processing section forgenerating a moving image of the second component having the temporaland spatial resolutions thereof increased, based on information on themoving image of the first color component, information on the movingimage of the second color component and the motion information.
 3. Theimaging and processing device of claim 2, wherein: the processingsection holds a relational expression of the moving image of the secondcolor component taken by the second imaging section and the moving imageof the second component having the spatial resolution thereof increased;and the processing section generates the moving image of the secondcomponent having the spatial resolution thereof increased, in accordancewith the degree at which the relational expression is fulfilled, using,as constraining conditions, temporal and spatial correspondence betweenthe moving image of the first color component and the moving image ofthe second color component, a condition for smoothness between pixelsclose to each other in the image, an assumption that the brightness ofan imaging subject moving in the image is constant, and a localcorrelation between pixel values of the first color component and thesecond color component.
 4. The imaging and processing device of claim 3,wherein the processing section outputs a moving image, which fulfillsthe relational expression best under the moving image of the secondcolor component taken by the second imaging section and the constrainingconditions, as the moving image of the second component having thespatial resolution thereof increased.
 5. The imaging and processingdevice of claim 1, wherein: the processing section holds a relationalexpression of the moving image of the second color component taken bythe second imaging section and the moving image of the second componenthaving the spatial resolution thereof increased; and the processingsection outputs a moving image fulfilling the relational expression asthe moving image of the second component having the spatial resolutionthereof increased, using, as constraining conditions, temporal andspatial correspondence between the moving image of the first colorcomponent and the moving image of the second component, and a conditionfor smoothness between pixels close to each other in the image.
 6. Theimaging and processing device of claim 2, wherein: the processingsection holds a relational expression of the moving image of the secondcolor component taken by the second imaging section and the moving imageof the second component having the spatial resolution thereof increased;and the processing section outputs a moving image fulfilling therelational expression as the moving image of the second component havingthe spatial resolution thereof increased, using, as constrainingconditions, temporal and spatial correspondence between the moving imageof the first color component and the moving image of the second colorcomponent, a condition for smoothness between pixels close to each otherin the image, and an assumption that the brightness of an imagingsubject moving in the image is constant.
 7. The imaging and processingdevice of claim 1, further comprising a third imaging section, wherein:the separation section separates the visible light into the first colorcomponent, the second color component and a third color component; thethird imaging section takes images which form a moving image of thethird color component with a third spatial resolution and a thirdtemporal resolution by exposure for a third charge accumulation timeperiod; and the third charge accumulation time period is shorter thanthe second charge accumulation time period, the third spatial resolutionis lower than the second spatial resolution, and the third temporalresolution is higher than the second temporal resolution.
 8. The imagingand processing device of claim 7, wherein: the processing section holdsa relational expression of the moving image of the second colorcomponent taken by the second imaging section and the moving image ofthe second component having the spatial resolution thereof increased;and the processing section outputs a moving image fulfilling therelational expression as the moving image of the second component havingthe spatial resolution thereof increased, using, as constrainingconditions, temporal and spatial correspondence between the moving imageof the first color component, the moving image of the second colorcomponent and the moving image of the third color component, and a localcorrelation among pixel values of the first color component, the secondcolor component and the third color component.
 9. The imaging andprocessing device of claim 7, wherein: the processing section holds arelational expression of the moving image of the second color componenttaken by the second imaging section and the moving image of the secondcomponent having the spatial resolution thereof increased; and theprocessing section outputs a moving image fulfilling the relationalexpression as the moving image of the second component having thespatial resolution thereof increased, using, as constraining conditions,temporal and spatial correspondence among the moving image of the firstcolor component, the moving image of the second color component and themoving image of the third color component, a condition for smoothnessbetween pixels close to each other in the image, an assumption that thebrightness of an imaging subject moving in the image is constant, and alocal correlation among pixel values of the first color component, thesecond color component and the third color component.
 10. An imagingdevice forming an image processing system together with an imageprocessing device, the imaging device comprising: a separation sectionfor separating visible light into at least a first color component and asecond color component; a first imaging section for taking a movingimage of the first color component, the first imaging section takingimages which form the moving image with a first spatial resolution and afirst temporal resolution by exposure for a first charge accumulationtime period; a second imaging section for taking a moving image of thesecond color component, the second imaging section taking images whichform the moving image with a second spatial resolution higher than thefirst spatial resolution and a second temporal resolution lower than thefirst temporal resolution by exposure for a second charge accumulationtime period longer than the first charge accumulation time period; and acontrol section for controlling imaging conditions of the first imagingsection and the second imaging section; wherein the imaging deviceoutputs information on the moving image of the first color component andinformation on the moving image of the second color component in orderto allow the image processing device to generate a moving image of thesecond component having the temporal resolution thereof increased. 11.The imaging device of claim 10, further comprising a third imagingsection, wherein: the separation section separates the visible lightinto the first color component, the second color component and a thirdcolor component; the third imaging section takes images which form amoving image of the third color component with a third spatialresolution and a third temporal resolution by exposure for a thirdcharge accumulation time period; and the third charge accumulation timeperiod is shorter than the second charge accumulation time period, thethird spatial resolution is lower than the second spatial resolution,and the third temporal resolution is higher than the second temporalresolution.
 12. An image processing device for receiving information ona moving image from an imaging device and processing the information,wherein: the imaging device includes: a separation section forseparating visible light into at least a first color component and asecond color component; a first imaging section for taking a movingimage of the first color component, the first imaging section takingimages which form the moving image with a first spatial resolution and afirst temporal resolution by exposure for a first charge accumulationtime period; a second imaging section for taking a moving image of thesecond color component, wherein the second imaging section taking imageswhich form the moving image with a second spatial resolution higher thanthe first spatial resolution and a second temporal resolution lower thanthe first temporal resolution by exposure for a second chargeaccumulation time period longer than the first charge accumulation timeperiod; and a control section for controlling imaging conditions of thefirst imaging section and the second imaging section; the imageprocessing device comprising: a motion detection section for receivinginformation on the moving image of the first color component andinformation on the moving image of the second color component andgenerating motion information based on a time-wise change of the movingimage of the first color component; and a processing section forgenerating a moving image of the second component having the spatialresolution thereof increased, based on the information on the movingimage of the first color component, the information on the moving imageof the second color component and the motion information.
 13. An imageprocessing method, comprising the steps of: receiving, from the imagingdevice of claim 10, the information on the taken moving image of thefirst color component and the information on the taken moving image ofthe second color component; generating motion information based on atime-wise change of the moving image of the first color component;obtaining a relational expression of the moving image of the secondcolor component and a moving image of the second component having aspatial resolution thereof increased; and generating the moving image ofthe second component having the spatial resolution thereof increased, inaccordance with the degree at which the relational expression isfulfilled, using, as constraining conditions, temporal and spatialcorrespondence between the moving image of the first color component andthe moving image of the second color component, a condition forsmoothness between pixels close to each other in the image, and anassumption that the brightness of an imaging subject moving in the imageis constant.
 14. The image processing method of claim 13, wherein thestep of generating the moving image of the second component generatesthe moving image of the second component having the spatial resolutionthereof increased, in accordance with the degree at which the relationalexpression is fulfilled, further using, as a constraining condition, alocal correlation between the pixel values of the first color componentand the second color component.
 15. An image processing method,comprising the steps of: receiving, from the imaging device of claim 10,the information on the taken moving image of the first color componentand the information on the taken moving image of the second colorcomponent; obtaining a relational expression of the moving image of thesecond color component and a moving image of the second component havinga spatial resolution thereof increased; and generating the moving imageof the second component having the spatial resolution thereof increased,in accordance with the degree at which the relational expression isfulfilled, using, as constraining conditions, temporal and spatialcorrespondence between the moving image of the first color component andthe moving image of the second color component, and a condition forsmoothness between pixels close to each other in the image.
 16. An imageprocessing method, comprising the steps of: receiving, from the imagingdevice of claim 10, the information on the taken moving image of thefirst color component and the information on the taken moving image ofthe second color component; obtaining a relational expression of themoving image of the second color component and a moving image of thesecond component having a spatial resolution thereof increased; andgenerating the moving image of the second component having the spatialresolution thereof increased, in accordance with the degree at which therelational expression is fulfilled, using, as constraining conditions,temporal and spatial correspondence between the moving image of thefirst color component and the moving image of the second colorcomponent, and a local correlation between the pixel values of the firstcolor component and the second color component.
 17. An image processingmethod, comprising the steps of: receiving, from the imaging device ofclaim 11, the information on the taken moving images of the first colorcomponent, the second color component and the third color component;obtaining a relational expression of the moving image of the secondcolor component and a moving image of the second component having aspatial resolution thereof increased; and outputting a moving imagefulfilling the relational expression as the moving image of the secondcomponent having the spatial resolution thereof increased, using, asconstraining conditions, temporal and spatial correspondence among themoving image of the first color component, the moving image of thesecond color component and the moving image of the third colorcomponent, a condition for smoothness between pixels close to each otherin the image, an assumption that the brightness of an imaging subjectmoving in the image is constant, and a local correlation among the pixelvalues of the first color component, the second color component and thethird color component.
 18. A computer program embodied in acomputer-readable medium for causing a processor included in an imageprocessing device to generate data on a moving image based on data on afirst moving image and data on a second moving image, the computerprogram causing the processor to execute the steps of: receiving, fromthe imaging device of claim 11, the information on the taken movingimages of the first color component, the second color component and thethird color component; reading a relational expression of the movingimage of the second color component and a moving image of the secondcomponent having a spatial resolution thereof increased; and generatinga moving image of the second component having a spatial resolutionthereof increased, in accordance with the degree at which the relationalexpression is fulfilled, using, as constraining conditions, temporal andspatial correspondence between the moving image of the first colorcomponent and the moving image of the second color component, acondition for smoothness between pixels close to each other in theimage, an assumption that the brightness of an imaging subject moving inthe image is constant, and a local correlation between the pixel valuesof the first color component and the second color component. 19.(canceled)
 20. The imaging and processing device of claim 2, wherein:the processing section holds a relational expression of the moving imageof the second color component taken by the second imaging section andthe moving image of the second component having the spatial resolutionthereof increased; and the processing section outputs a moving imagefulfilling the relational expression as the moving image of the secondcomponent having the spatial resolution thereof increased, using, asconstraining conditions, temporal and spatial correspondence between themoving image of the first color component and the moving image of thesecond component, and a condition for smoothness between pixels close toeach other in the image.
 21. The imaging and processing device of claim2, further comprising a third imaging section, wherein: the separationsection separates the visible light into the first color component, thesecond color component and a third color component; the third imagingsection takes images which form a moving image of the third colorcomponent with a third spatial resolution and a third temporalresolution by exposure for a third charge accumulation time period; andthe third charge accumulation time period is shorter than the secondcharge accumulation time period, the third spatial resolution is lowerthan the second spatial resolution, and the third temporal resolution ishigher than the second temporal resolution.